Advanced Searching

This document is part of an guide to using the Matrix Requirements SDK (github page here, and part one of this guide here). We'll continue working in the Node environment established in part one.

We've already seen the method Project.searchForIds() which takes a query string and returns a list of Item IDs. But we may want to execute more sophisticated searches, and for this we have two methods:

searchForItems(term, filter, treeOrder, mask) which returns an array of Item objects, and
searchRaw(term, options) which returns an array of ISearchResult objects.

The methods do the same thing, but searchForItems is designed to be friendlier to work with.

Searching with a mask

We'll create a mask that gives us back partial Items with only one field brought down from the server:

partial-search-1.js

const assert = require("assert");
const lib = require("./lib.js");

async function run() {
    const server = await lib.getServerConnection("clouds5");
    const wheely = await server.openProject("WHEELY_OBSERVABLE");
    // Construct a mask that includes fields, but no labels or up or down links.
    let mask = wheely.constructSearchFieldMask({ includeFields: true });
    const catREQ = wheely.getCategory("REQ");
    const descriptionFieldId = catREQ.getFieldIdFromLabel("Description")[0];
    mask.addMask(catREQ, [descriptionFieldId]);

    // Now bring down all Items of category REQ with this field mask.
    const reqs = await wheely.searchForItems("mrql:category=REQ", "", false, mask);

    let rows = [];
    for (let i = 0; i < Math.min(5, reqs.length); i++) {
        const item = reqs[i];
        assert(!item.hasAllFields());
        assert(item.hasFieldId(descriptionFieldId));

        const id = item.getId();
        const title = item.getTitle();
        const descriptionField = item.getSingleFieldByName("Description").getHandler();
        const description = descriptionField.getHtml();
        rows.push(`<tr><td>${id}</td><td>${title}</td><td>${description}</td></tr>`);
    }
    console.log(`<table>${rows.join('\n')}</table>`);
}

run().then(() => process.exit(0));

Note that we asserted that the Items brought down only have the one Category REQ field we asked for, "Description". Typical HTML output is below:

REQ-1

Design / Looks

The wheelchair should not look like a medical device but like something which looks nice to kids from 6 to 10 years.

REQ-2

Sized for Kids

The wheelchair should fit all standard heights and weights for kids from 6 to 12 years of age.

3 to 15 Years

Boys	Average Values		Normal Range
Age (Years)	Weight (Pounds)	Length (Inches)	Weight (Pounds)	Length (Inches)
2	27.5	34.2	22.8-33.0	31.7-36.3
3	31.0	37.5	26.1-38.5	35.2-39.8
4	36.0	40.3	29.0-44.0	37.5-43.2
5	40.5	43.0	33.0-52.5	39.8-45.7
6	45.5	45.5	36.5-59.0	42.2-48.6
7	50.5	48.0	40.5-68.0	44.5-51.3
8	56.5	50.4	45.0-77.0	46.7-54.3
9	63.0	52.5	49.5-88.0	48.7-56.5
10	70.5	54.5	56.0-100.5	50.5-58.8
11	78.5	56.5	60.5-114.0	52.0-61.0
12	88.0	58.7	66.5-130.0	54.0-63.5
13	100.0	61.5	74.5-144.0	56.3-66.6
14	112.0	64.5	84.0-159.5	59.1-69.7
15	123.5	67.0	92.5-172.5	61.6-71.7

REQ-3

Wheels

There must be 4 wheels makes it better

REQ-4

Seat

There must be a seat which

is comfortable for extended periods of sitting (> 8 hours)
can be cleaned easily

REQ-5

Tablet / Ipad Holder

The wheelchair must allow to attach a table from 7" to 11"

Masking across multiple Categories

What if we are looking for the Description field if we have a REQ Item, and the Use Case Steps field if we have a UC item? No problem! You can construct the mask so that it will apply correctly to each kind of item returned:

partial-search-2.js

const assert = require("assert");
const lib = require("./lib.js");

async function run() {
    const server = await lib.getServerConnection("clouds5");
    const wheely = await server.openProject("WHEELY_OBSERVABLE");

    // Construct a mask that includes fields, but no labels or up or down links.
    let mask = wheely.constructSearchFieldMask({ includeFields: true });

    const catREQ = wheely.getCategory("REQ");
    const descriptionFieldId = catREQ.getFieldIdFromLabel("Description")[0];
    mask.addMask(catREQ, [descriptionFieldId]);

    const catUC = wheely.getCategory("UC");
    const ucStepsFieldId = catUC.getFieldIdFromLabel("Use Case Steps")[0];
    mask.addMask(catUC, [ucStepsFieldId]);

    // Now bring down all Items of category REQ and UC with this field mask.
    const reqs = await wheely.searchForItems("mrql:category=REQ or category=UC", "", false, mask);

    let reqCount = 0, ucCount = 0;
    for (let i = 0; i < reqs.length; i++) {
        const item = reqs[i];
        assert(!item.hasAllFields());

        if (item.getCategory() == catUC) {
            assert(item.hasFieldId(ucStepsFieldId));
            ucCount++;
        } else if (item.getCategory() == catREQ) {
            assert(item.hasFieldId(descriptionFieldId));
            reqCount++;
        }
    }
    console.log(`Project has ${reqCount} REQ Items, and ${ucCount} UC Items`);
}

run().then(() => process.exit(0));

Run this program for the following result:

mstanton@darkstar:~/examples/users-guide (main)$ node partial-search-2
Project has 22 REQ Items, and 3 UC Items
mstanton@darkstar:~/examples/users-guide (main)$

Category object also returns search results

If you want to get all the items for a particular Category, you can request this directly from the Category object without using one of the search methods already mentioned, see line 15 below. And you can still use a mask to retrieve partial items if you like.

partial-search-3.js
const assert = require("assert");
const lib = require("./lib.js");

async function run() {
    const server = await lib.getServerConnection("clouds5");
    const wheely = await server.openProject("WHEELY_OBSERVABLE");

    // Construct a mask that includes fields, but no labels or up or down links.
    let mask = wheely.constructSearchFieldMask({ includeFields: true });

    const catUC = wheely.getCategory("UC");
    const ucStepsFieldId = catUC.getFieldIdFromLabel("Use Case Steps")[0];
    mask.addMask(catUC, [ucStepsFieldId]);

    const UCs = await catUC.getItems({ mask });
    let total = 0;
    for (let uc of UCs) {
        assert(!uc.hasAllFields());
        assert(uc.hasFieldId(ucStepsFieldId));

        const fieldHandler = uc.getFieldById(ucStepsFieldId).getHandler();
        total += fieldHandler.getRowCount();
    }
    console.log(`The average number of Use Case Steps in UC Items is ${total / UCs.length}`);
}

run().then(() => process.exit(0));

And the output follows:

mstanton@darkstar:~/examples/users-guide (main)$ node partial-search-3
The average number of Use Case Steps in UC Items is 5.666666666666667
mstanton@darkstar:~/examples/users-guide (main)$

A search retrieving Label information

Of course, maybe you don't want any Fields to be brought down at all, only, say, labels. It's all in the constructSearchFieldMask() function which takes a helper object:

  * includeFields      default true
  * includeLabels      default true
  * includeDownlinks   default false
  * includeUplinks     default false

  For example:
    mask = constructSearchFieldMask({ includeFields: false, includeUplinks: true });

Let's have a code example where we get Item objects, but no Category Fields, thus reducing download time. We get all the labels and print out the ones we found:

partial-search-4.js

const lib = require("./lib.js");

async function run() {
    const server = await lib.getServerConnection("clouds5");
    const wheely = await server.openProject("WHEELY_OBSERVABLE");

    // Construct a mask that only includes labels.
    let mask = wheely.constructSearchFieldMask({ includeLabels: true });

    const catUC = wheely.getCategory("UC");
    const ucStepsFieldId = catUC.getFieldIdFromLabel("Use Case Steps")[0];
    mask.addMask(catUC, [ucStepsFieldId]);

    const items = await wheely.searchForItems("mrql:category=TC", "", false, mask);
    let foundLabels = new Set();
    for (let item of items) {
        for (let label of item.getLabels()) {
            foundLabels.add(label);
        }
    }
    let output = [];
    for (let l of foundLabels.values()) output.push(l);
    console.log(`Found the following labels on TC Items: ${output.join(", ")}`);
}

run().then(() => process.exit(0));

And the output:

mstanton@darkstar:~/examples/users-guide (main)$ node partial-search-4.js
Found the following labels on TC Items: ORANGE, NIGHTTIME, FORPRINTING, APPLE
mstanton@darkstar:~/examples/users-guide (main)$

Cool, we can see our Labels on TCs.

"Raw" (low-level) search results

If you want, you can go more "old school" and use method searchRaw() which takes parameter fieldList. You can use the mask as we've described to come up with your field mask, and then you can ask the mask for it's fieldMaskString in order to fill in that parameter correctly. Note that the field IDs are referenced, because those are unique:

partial-search-5.js

const lib = require("./lib.js");

function getMaskString(project) {
    // Construct a mask for the purposes of getting a mask field string
    let mask = project.constructSearchFieldMask({ includeFields: true });
    const catREQ = project.getCategory("REQ");
    mask.addMaskByNames(catREQ, ["Description"]);
    const catUC = project.getCategory("UC");
    mask.addMaskByNames(catUC, ["Use Case Steps"]);
    return mask.getFieldMaskString();
}

async function run() {
    const server = await lib.getServerConnection("clouds5");
    const wheely = await server.openProject("WHEELY_OBSERVABLE");
    const maskString = getMaskString(wheely);
    let searchResults = await wheely.searchRaw("mrql:category=REQ or category=UC", "", maskString);
    for (let result of searchResults) {
        const strValue = JSON.stringify(result);
        console.log(`${strValue.substring(0, 60)}...`);
    }
}

run().then(() => process.exit(0));

The output of searchRaw isn't wrapped into Item objects, but remains in a low-level type:

mstanton@darkstar:~/work/matrix-sdk/examples/users-guide (main)$ node partial-search-5
{"itemId":"REQ-1","version":3,"title":"Design / Looks","down...
{"itemId":"REQ-2","version":3,"title":"Sized for Kids","down...
{"itemId":"REQ-3","version":24,"title":"Wheels","downlinks":...
{"itemId":"REQ-4","version":1,"title":"Seat","downlinks":[],...
...

Continuous Log of REST requests

The SDK also has information about the REST calls made to the server over time. There is a list of the calls made. You might use this list to verify that only one call was made for a powerful search request.

search-fetchlog.js

const lib = require("./lib.js");

lib.getServerConnection("clouds5").then((server) => {
    server.openProject("WHEELY_OBSERVABLE").then(async (project) => {
        project.searchForItems("mrql:category=REQ or category=UC").then((items) => {
            console.log(server.getFetchLog().join("\n"));
        });
    });
});

The code above was written in the classic Node style using Promises rather than async/await just for fun. After running, you can see 3 calls were made to the server:

mstanton@darkstar:~/examples/users-guide (main)$ node search-fetchlog
https://clouds5.matrixreq.com/rest/1/
https://clouds5.matrixreq.com/rest/1/WHEELY_OBSERVABLE?adminUI=1
https://clouds5.matrixreq.com/rest/1/WHEELY_OBSERVABLE/needle
mstanton@darkstar:~/examples/users-guide (main)$

One for Server information. The second call for Project information. The final request ("needle") is the search. We were finding needles in haystacks today!

Have a look at Part 3 of the User's Guide for more information on saving items, uplinks and downlinks between items, labels and attachments. Thanks for your time!