297 — Deep Search

Ultimheat Museum

Notes (June 2023) 233-map

[233-map][https://github.com/blueink-bkk/233-mapp]

$ docker ps =>
mup-249-deep-seach:latest  Up 2 years  80/tcp, 0.0.0.0:32050->3000/tcp   249-deep-seach

.mup-deploy/249-deep-search-dallas-32050 =>  ~/2020/249-deep-search-yapp

~/2020/233-map was once museum app ???
~/2022/279-mapp-svelte

dkz@LG:~/2023/249-deep-search-yapp$ git remote -v
github	https://github.com/blueink-bkk/233-mapp.git (fetch)
github	https://github.com/blueink-bkk/233-mapp.git (push)

Server:

mup-249-deep-seach seems to be the one running.

docker images
REPOSITORY               TAG            IMAGE ID            CREATED             SIZE
mup-233-mapp             build          9fa4293c7ea2        9 months ago        657MB
mup-233-mapp             latest         9fa4293c7ea2        9 months ago        657MB
mup-231-blueink-api      build          da1021dfeea6        2 years ago         816MB
mup-231-blueink-api      latest         da1021dfeea6        2 years ago         816MB
mup-231-blueink-api      previous       1987b2ae9b48        2 years ago         816MB
mup-249-deep-seach       build          aa066d5ee99c        2 years ago         724MB
mup-249-deep-seach       latest         aa066d5ee99c        2 years ago         724MB
mup-249-deep-seach       previous       b406f968f291        2 years ago         724MB
mup-dico-chauffage       latest         9c9506e8047e        3 years ago         569MB
mup-museum-admin         latest         893b2e03d0f6        4 years ago         587MB

NGINX on BlueInk

PING ultimheat.com (101.32.27.53) Hong-Kong
PING museum.ultimheat.com (45.33.5.114) Linode Blueink Dallas

RESULTS:

https://github.com/blueink-bkk/233-mapp.git is working. (June 2, 2023)
dkz@LG:~/2021/233-mapp is sync with github.
change url in .env.json
~/.mup-deploy/249-deep-search-dallas-32050 is deploying on 32050.

Notes:

80K documents (PDF file) with 1-300 pages.
each document has 1 or more authors
each document is related to a constructor / manufacturer
each manufacturer has labels (brands / marques).
each document has a title. Not sure unique.
possible same label for different constructors - when merging companies
possible same author for different constructors - when merging companies
search on labels, constructors, authors, title (document), or full text.

User Interface

Welcome page
quick-search --> /client/quick-search --> Meteor.call get-itemx, museum-index
deep-search --> /client/deep-search --> Meteor.call deep-search, pdf-pages-count
index-auteurs --> /client/auteurs --> Meteor.call auteur-info, new-auteur
index-constructeurs --> /client/index-constructeurs --> Meteor.call list-articles
index-marques --> /client/index-marques --> Meteor.call list-articles
index-titres (S3) --> /client/index-s3 --> Meteor.call list-articles

Architecture (233-map)

Uses "massive" on tables adoc.page, adoc.file, adoc.pagex
for all queries: lpath <@ 'museum.md'
Not sure we are still using plv8.
nspell used for deep-search_v1
sql procedure : /home/dkz/2022/236-xlsx-museum-preflight/sql

Database

adoc.file

id:         primary key (unique sequential)
xid:        external uid
path:       ltree -ex: 'museum.md.c.2'
lang:       fr, en
data:       jsonb document
mtime:      modified time

adoc.page

file_id: => adoc.file.id
pageno:     integer
raw_text
tsv:        text-search-vector

adoc Sql procedures

adoc.commit_file (_path ltree, _xid text, _data jsonb, _lang text default 'fr'::text) returns integer
adoc.new_file (__path text, _xid text, _lang text default 'fr'::text) returns integer
adoc.new_file (ltree, text, jsonb, text) cascade;
adoc.write_page (_path text, _xid text, _pageno integer, _raw_text text) returns void
adoc.write_pagex (_path text, _xid text, _pageno integer, _data jsonb, _raw_text text) returns void
adoc.page_vector_update () returns trigger
adoc.search_pages_rank_cd2 (path ltree, query text) returns table
adoc.remove_xid (ltree, xid)

adoc Triggers

trigger tsvector_update BEFORE INSERT OR UPDATE ON adoc.page FOR EACH ROW EXECUTE PROCEDURE tsvector_update_trigger('tsv', 'pg_catalog.french', 'raw_text');

Methods (13)

list-articles(db,id) -> list-articles.js
list-catalogs
assets-folder-directory(folder) => directory. Uses Linux fs-xattr.getSync
deep-search(lpath, ts_query) => db.adoc.search_pages_rank_cd2()
get-itemx(xid) : find item based on xid (external-id or custom) not primary key
get-pub-directory : pub/advertising -- obsolete
index-auteurs // auteurs, w/ articles done on server/plv8
index-constructeurs // list all catalogs, then aggregate on client(constructeur)
index-marques // marques, w/ articles done on server/plv8
index-s3: from table cms_articles__directory
museum-index: from table adoc.pagex
pdf-pages-count: from table adoc.file
upload-file(file, fileData) : uses xattr.setSync(file, attribute)

HTTP-SERVER

Express w/ (express-device)
Busboy: parsing incoming HTML form data. 8M downloads weekly.

SSR.compileTemplate (Assets.getText)

page-template => views/page-template.html
xref-constructeurs => views/xref-constructeurs.html
page => views/page.html
catalogs => views/catalogs.html
phone-welcome => views/phone-welcome.html

ROUTES (app.get)

/xref-constructeurs
/catalogs
/page/:id
/ping
/pub-tester2
/upload2
/sitemap.txt

ROUTES (app.post)

/gateway/:fn

nspell

80K words for French.

Uploading Documents

2022/236-xlsx-museum-preflight

Programs

101-xlsx2json.js xlsx to json file, with validation on auteurs, constructeurs
102-json2md-files.js -- Using load-json-file, write-json-file.
103-split-pdf.js -- pdfjs-dist, massive, find. Split and load pdf pages from MD files.
104-upload-pdf-pages.js -- klaw, massive, walkSync, db.adoc.write_page.
107-upload-xlsx.js -- raw-text, and article attributes for xdata/jsonb
108-index-marques.js -- call to /lib/index-des-marques.js
109-index-auteurs.js -- massive, sql.
110-index-articles-s3.js -- massive
112-index-constructeurs.js -- massive
reformat.js -- trim, cleanup, country names to ISO.
xlsx2json.js -- xlsx, jsonfile.

Sql Stored Procedures

101-create-tables.sql -- tables: adoc.file, adoc.page, -- procedures: new_file, commit_file, new_file, write_page, write_pagex, search_pages_rank_cd2
102-test-plv8.sql -- Obsolete
103-index-marques.sql -- using plv8 -> Obsolete
104-list-pages.sql : adoc.list_pages(lpath, xid)
105-pagex-view.sql : adoc.pagex as a sql-View
106-fix-xid.sql : copy xid into jsonb.xid
107-list-files.sql : adoc.list_files, adoc.list_filex
108-remove-path.sql : adoc.remove_path(ltree) -> remove files in ltree.

RETROFITTING 233-MAP-2023

still using Blaze.
removing _assert in server
removing Google Analytics - slowing down.
deep-search on "jules cesar jules" Not good.

alternative DB access.

new: /server/db-pool.js using env and settings.

Text Search

To be used first for project documentation (lekkyweb). Scan S3 files and load/update tsv
Need to scan URL from Openacs FileStorage.
We have control of lekkyweb files -> use semantic tags: main, etc...
Should have tags: article, section, etc...
lekkyweb.com : All files on S3 are on LG://2022/292-lekkyweb/blogs. Each md-file is converted into html using fw4b. See documentation 281-watcher-4-bucket

css detected: