Replies: 6 comments 22 replies
Minutes for Oslo meeting (February 8, 2024)
|
|
Summarizing the technical meeting on Stage 2 today. We now have the implementation of 2 steps from stage 2, langid and text extraction with boilerplate removal. Both run in parallel across several LUMI nodes, and across input lines on each node. We have agreed on the next TODOs:
The slides are here (work in progress, will be updated, the link allows to leave comments there): |
|
Some principle decisions to make:
|
|
Returning to our meeting today, two samples of HTMLs and texts, for all CC crawls and all wide crawls, are commited to https://github.com/hplt-project/warc2text-runner/tree/main/two/sample100/CC and https://github.com/hplt-project/warc2text-runner/tree/main/two/sample100/wide. All data for these samples is in https://github.com/hplt-project/warc2text-runner/tree/main/two/sample100/*.zst. Also I've checked a couple of questions raised.
|
|
Digging into the example mentioned by @ZJaume (https://github.com/hplt-project/warc2text-runner/blob/main/two/sample100/CC/27.txt), I found that by default Trafilatura falls back to some simple baseline method that leaves lots of boilerplate in case the main method produces too short text. So basically for this example we see the text extracted by the baseline, and the main method extracts nothing. My intuition is that we don't have much really valuable running text on this page, so extracting nothing may be better for this page rather than leaving all the boilerplate (giving a bit lower recall but much higher precision when considering the task of extracting "running text in natural language"). More details on the slides, starting from slide 11: https://docs.google.com/presentation/d/1-AUBkFBwuPXUPMY756FyXhmFHr3sv3vm-dFJ7qG9Yq4/edit?usp=sharing Thus, I played with Trafilatura settings and for sample100 commited 2 new versions of extracted texts, using Trafilatura without fall back to the baseline (.MIN_EXTRACTED_SIZE_0.txt) and using Trafilatura without falling back to the baseline but employing readability-lxml and justext in addition to the main extraction algorithm (.no_fallback_False.txt). The last setting results in 2x slower processing though. Comparing basic statistics for 4 methods (warc2text and 3 versions of Trafilatura): Without falling back to the baseline, Trafilatura returns empty texts for 20-40% of webpages. But looking at those pages it seems to me much more reasonable result than the baseline or warc2text gives if we want to extract running texts, not everything. Also it gives ~6-7x fewer lines, and ~3-4 fewer bytes, but ~2x longer lines on average. So looks like it removes lots of short lines that constitute a significant part of the corpus and probably contain lots of boilerplate. I see 2 reasonable ways to proceed:
|
|
Following our today discussion with @oepen, @akutuzov and @davda54, this is how one can compare a sample of webpages processed with different Trafilatura settings. After cloning https://github.com/hplt-project/warc2text-runner/tree/main: Choose C among CC and wide to switch between crawls, and change X to switch between sampled web pages. The selected html page will be opened in a web browser, and 2 versions of text extracted from this page with different settings will be shown in the terminal. To switch between tabs in vimdiff: ctrl+w <right/left arrow> |


Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
In early February, we are ready to launch our second end-to-end iteration, i.e. run our revised data processing (aka text extraction) pipeline on lots more data, train new sets of language and translation models, and evaluate these appropriately. This discussion is intended to decide on the technical setup and schedule for the data processing part.
We are now looking at two common stages, at times called
warc2htmlandhtml2text, which prepare the raw text collections for either monolingual or bilingual downstream processing. These processes constitute the third stage, called monotexting and bitexting in HPLT. Stage #1 will run close to the storage, i.e. on the dedicated NIRD and CESNET nodes that are available to the project. For this stage, we still need to decide the batch size, i.e. how many WARC files eachwarc2htmlis to process. For each such process, there will be a pair of (line-aligned) resulting output files, one of the HTML, another one for all the metadata. The sum of these files will be much smaller than the original WARC collections and will form the input to Stage #2, which we expect to run on LUMI.The batch size should be chosen to make it practical and efficient to perform Stage #2. We have yet to build the Stage #2 tool, but we assume (i) it will be in Python, (ii) perform some URL-based filtering, (iii) call out to Trafilatura for boilerplate removal, (iv) invoke FasterText for language identification, (v) perform sharding to support bitexting, and (vi) write out a suitable number of files in a suitable format (probably bifurcated by document language). Based on contrastive benchmarking by Nikolay, we expect Stage #2 to be 20–30 more computationally expensive than the original
warc2text. In other words, we are making a substantive investment in terms of CPU hours here.In terms of storage, we expect that our roughly 4,5 pib of WARC files will require at least some 500 tib of storage after Stage #1.
A rough calculation from a joint bus ride between Skeikampen and Oslo estimates the cost of Stage #1 to around 350,000 CPU hours, and up to some 10,000,000 CPU hours for Stage #2. The latter is embarrassingly parallel. To run it effectively on LUMI, we might want to break it into jobs that on average correspond to one single-node CPU day, which is only about 1600 jobs. If each job processes one input file and parallelizes internally (e.g. using 256 Posix threads), it might seem sufficient to aim for Stage #1 to produce just a few thousand (pairs of) raw HTML files. What would be the average estimated file size on this view?
And what about the output from Stage #2. Ideally, we bifurcate by language (which may multiply our file count by about 200) at this point and shard in a manner suitable for bi-texting (which, in turn, will reduce the overall number of files). How does sharding work internally, and can Stage #2 perform it directly, while going from raw HTML to raw text?
All reactions