314 — TSV lekkyweb.com

Files

• 101-scan-folder.js : list files in folder
• 102-scan-html-file.js : parse html with node-html-parser. PB with 1 file.
• 103-scan-Files-cheerio.js: parse html file with cheerio.
• 104-scan-s3-bucket.js: parse html files from bucket - squeeze using cheerio.
• squeeze.js : extract body, remove section.tsv-ignore

Architecture - Requirements

T/S for GIS data.

• 64K cities in Thailand.
• each with some data html, MD, yaml.
• find a city and get location/data.
• each city is a row in table 'article' => raw-text, tsvector
• must have link to geometry table (or) have column geometry ??
• oacs-geo-features : (object_id, label, owner, geometry, ltree)
• fts.article : (raw-text, tsvector, ltree, jsonb, )

T/S Articles:

Fusion with Postgis geometry. If geometry is irrelevant for some articles, just leave the column undefined.

• Deep search is done on table fts.articles.
• Each 'article' has raw-text, tsvector and reference to original text file.
• Could have link to picture, or collection of pictures (why not).
• Original text file is most likely stored in S3-bucket but could be cr_revisions table
• Articles have different origins: PDF, HTML, JSON, YAML
• Each article has a reference to acs_objects table.
• Queries on table fts.articles need to use a join to other tables.
• Queries on html files structured in articles, could give href w/hash
• For a table with reference to PDF do we need with classification in ltree.
• when document is updated, all pages/articles must be removed first. Then new pages are created. The content to be displayed is pdf:pageNo or html:href.

Case: PDF files

• Queries on PDF pages will do join with PDF files tables or content-repository.
• Each PDF page is like an article: we need direct access to that page.
• All PDF pages have the same lpath.
• THIS IS OBSOLETE IN 2023 : In case of PDF pages, each page is a PDF ! see pdf-split, or we use browser/server capability to fetch only 1 page from the pdf.
• Queries will be customised for each application. T/S on PDF will be coded with join on table PDF documents: because each article is in fact a page. But in that case, no need for GIS data.
• Seach on PDF files has no connection with GIS. Should be treated differently: PDF files should be URL in oacs content-repository, or a stand alone table.

photo-album:

• multiple photos for an article (ex incident, monument)
• photos are stored in a vector in the article.

  • we need a jsonb in the article. NO !!

  • and a jsonb in the document.

    (OR) monument/incident is a document AND: list of pictures is stored in the document.

ftsv_articles

New architecture:

• is not a page (PDF)
• cols: object_id, raw_text, tsvector, geometry, ltree, url
• join with acs_objects to get a jsonb.
• url: reference to original document.
• jsonb: to store pictures, etc...
• one source could generate many articles (gis/cities). Do we need the concept of batch? to remove a group of articles.
• This last point is about multi dimensional classification.
• Or it's enough to link to a 'document' object, or concept of Collection.
• Can we use acs_object.context for that ?

Batch features - Collections

• Most articles are authored separately. This is not the case for GIS features. They might be imported and updated in batches. We need a table that links all the items in the collection together, like the pages of a PDF file. There is a need for a column, giving access to the collection or batch. That column could be undefined in some case. Looks like we have a folder here to represent collection, batch, etc...
• Should we create a cr-revision for each gis feature. It has a link to a folder.

Storing gis feature into a cr-revision.

• need to add jsonb column, and geometry. Why not.