316 product cards into Word in one day
An archive of 318 folders — products from a UK coffee-equipment store. Each folder needed the description, specs and reviews copied over from the website, and into an old .doc at that. The client was doing it by hand, five minutes per card. He finished two files and stopped: the rest meant another 26+ hours of copy-paste. I wrote a parser, and two hours later all 316 files were done.
What was broken
The client’s process was simple: open a product on the site, copy, paste into Word. He did two files that way and stopped. With good reason:
- The store threw a captcha even when you just switched tabs on the same product page.
- Word pasted the copied text as tables. The client worked around it via .txt: paste there first, then into Word. Double the work.
- Every product has its own numbered folder: a .txt with the link, a .doc for the text, an Excel file and photos. Only the .doc may be touched.
- Word 97-2003 format only. Not .docx — the old .doc.
What each file had to contain
The client sent the layout as seven screenshots. In order: breadcrumbs, title, price, return guarantee, Perfect for, description, list of drinks, specs (both the visible part and the full table from the Specifications tab), instructions, reviews. Reviews had their own rule: the oldest ones, from the last three pages, without replies.
One day for everything.
The solution
The first thing I decided: only the browser goes to the site, like a regular visitor. Parsing, formatting and writing to Word happen on the computer, and the site knows nothing about it.
The site only sees a regular visitor session. Parsing and writing to Word happen locally.
Python goes through the archive twice. First it pulls out the links, then it puts the collected data back into the same folders.
- Generator. Walks the folders from a given number, takes the product link from each .txt (skipping the category line) and builds collect.js from them. PowerShell drops the script straight onto the clipboard — you just paste it.
- Collector. JavaScript that runs in the browser console, the place where you can run code right on the open page. It takes each product in turn and pulls all its reviews from the store’s public API: the same address the site itself uses to load reviews onto its pages.
- A 1.5–3 second pause between products; reviews come in batches of 100. If a request fails, the script retries three times, waiting longer each time. The site isn’t hammered, so it has nothing to react to.
- Processor. Takes the newest data.json from Downloads, parses the pages with BeautifulSoup and lays out the text in the order of the layout. Replies to reviews get dropped: they have no rating.
- I write to Word via COM. Put simply, the program drives the installed Word as if a person with a mouse were sitting at it. The text goes straight into N.doc and is saved as 97-2003. No tables, because what goes in is clean text, not a chunk of web page.
- No Word? LibreOffice takes over. File open in Word? The script stops and says what’s wrong instead of breaking it.
- At the end, report.csv with a row per folder: title, how many drinks and specs, how many reviews on the site and how many were taken. No title or description means a “check manually” status.
A run takes two commands. The first generates the collector, the second processes what it collected.
Development story
The captcha
I told the client the script doesn’t click through tabs, so there’d be no captcha. He didn’t get it and asked me to show him. That same evening I recorded a video: three products from three different categories, the product page and the finished .doc side by side. No more questions after that.
Formatting
The client planned to finish the formatting by hand, because he didn’t believe the parser could handle it. It could: font sizes, bold headings, blank lines between blocks, all as in his sample. And blocks the product page doesn’t have simply don’t end up in the file.
What I took away
- When a client doesn’t get a technical explanation, explaining it again is pointless. Better to record a video. Three products on screen did more than my paragraph about tabs.
- A report per item is always worth it. In a minute, report.csv showed a card without a description and 11 duplicates the client never suspected.
- I didn’t have to get around the site’s protection at all. A normal browser session plus pauses, and the captcha never showed up once.
Result
The same folders now hold 316 filled .doc files, with report.csv next to them summarising every product. From archive to result took about two hours.
The parser stayed with the client. It’s tuned to this store and runs with two commands, so refreshing reviews or collecting another category is now a matter of minutes.