Fetching data and filtering best practices

[khairul]

Hi weweb community!

I have a question, since it is better to filter data in backend, how can I filter them by a variable in a page?

For example, I have a search input in my page, I want to fetch only my search input value from the backend. Does putting a filter in the 'fetch collection" page in weweb does that or it filters in the frontend?

I think it would be more secure and faster if we only fetch what we need from the backend, is that correct? I need your input and help. Thank you.

[aurelie]

You are correct, filtering data on the backend is often better. Its always a tradeoff

• Filtering on backend: you load less data (if you filter do not change often), and can have better filter on your database. Cons: you will have to wait the response of your backend to display the info
• Filtering on frontend: you load all the data (which can be fine if your dataset is not too big). The filtering will be more “instant”. Cons: do not use if the data need to be secure or if your dataset is too big.

How to do it:
You will probably have a search input somewhere, with a corresponding internal variable available.

• You will then bind your collection query parameters to this variable. Depending on your backend this can be a query params or something else.
• You need to add a onChange workflow on your input, and fetch your collection here

Two warnings:

• On first load, your search variable will be null. You need to decide what filter/param you pass to your backend in this usecase. We often have customer with strange behavior at initialisation because of this (variable not defined)
• If you are using a text input for your search (and not a select of a checkbox), please use the search input and not a normal text input. The search input provide what we call a “debounce” value, so the “onChange” event is triggered only if the value does not change for a delay you can configure. If you use a normal input, your backend will be requested for each letter entered, which is not really performant

Hope it helps

[khairul]

Sorry I don’t quite understan the binding to query parameters. If I add a filter at the fetch collections section, will it work the same? When I filter data in the collections page, does it fetch all the collection from backend or the filtered one only?

[caffeinatedwes]

@aurelie

The search input provide what we call a “debounce” value, so the “onChange” event is triggered only if the value does not change for a delay you can configure. If you use a normal input, your backend will be requested for each letter entered, which is not really performant

Thanks for mentioning this, @aurelie. I was just thinking about this functionality the other day.

To clarify, 1) Are you referring to the elements in the screenshot below?

and 2) How would I set this up? I couldn’t find a debounce value in the component, so I imagine this workflow setup (below) would still be triggered for each letter entered.

[aurelie]

Hi @caffeinatedwes,
a “Time delay” is very different from a debounce: time delay = will be executed with a delay, but all will be executed. Debounce = only the last value if no change during delay will be executed.

It is the “Search bar” element, but be carefull to select the search (which contain a text input). You can click on the small blue magnify icon, or use the navigator. Then you will see the delay, and will be able to connect to the onChange of it.

[aurelie]

It will depends which data source you are using. Which one are you using?

[caffeinatedwes]

Ahh, found it. Thank you!

Adding the screenshot and recapping for others.

To help others configure the desired functionality of not constantly calling the API with every character change, you would…

1. configure the delay under the component’s specific settings and then
2. set the workflow to trigger on the search bar with a “on change” trigger

That way, If you set the delay on the component to be, say, 1000 ms and constantly change the field (say every 300ms), your workflow would only run every 1000 seconds.

Correct me if I’m wrong, @aurelie

Related, perhaps for @Joyce—it would be really cool to have the delay feature on other elements. For example, I’m currently building CRM-like functionality and there would be a notes section. It would be great to allow the user to take all the notes they want without having to worry about clicking “save” or “update record” by adding in a similar delay functionality to the “long answer” input.

[aurelie]

Debouced means = we wait 1000ms without any change before sending the event.
1000ms is a bit too long. An usual value for this is between 300ms and 600ms, so it will let the user typing and only request when he paused the typing for watching result.
It can indeed be very usefull for autosave feature (we have it inside weweb editor to have history and backend update by “batches” for example).
I let Joyce handle the product part .

I think we can maybe do it with a bunch of JS, but that will not be really confortable. In the meantime using the “Search input” as a text field can work? If i remember correctly it is fully customizable as it contains a “normal” input.

[caffeinatedwes]

Thank you, @aurelie! Really helpful insight regarding the length of time for the debounce configuration.

Some additional info for @Joyce if it’s helpful: using the search field like @aurelie describes might work for for use cases, but the main limitation is that it doesn’t support multiple lines, which eliminates many use cases.

[khairul]

Hi @aurelie, what if I want to use the debounce feature for say a number input, how can I do that? Maybe I can add some code to existing search input or number input. I need to enable users to update their data live( I use supabase) and update their backend after the delay. Also can you enable binding for the placeholder for the search input? That would also help my case, thank you

[caffeinatedwes]

Related, I’m trying to implement that notes feature I discussed. It would be really helpful to have the debounce feature on the short text and long text input.

I feel like my options otherwise are pretty limited. I might have to add a save button (which defeats the purpose of the seamlessness of the debounce feature), for example.

Any suggestions, @aurelie?

[caffeinatedwes]

Would it be possible to implement debounce with custom JS?

[caffeinatedwes]

Actually, I’m thinking this is actually possible with WeWeb’s workflow builder.

Do you see any potential problems with this, @aurelie?

Perhaps the “on change” action triggers a workflow which…

1. Checks if there is a “debounce” variable. If there is, stop workflow.
2. If there is not a debounce variable, set one, have it expire in 400ms, and continue the workflow.
3. Make the API call.

This way, every change triggers the workflow, but if the user is typing rapidly, the workflow will be triggered with every character change, but the API call will only happen once every ~400ms.

Update:

I couldn’t get the above to work smoothly or without being overly complex, so I ended up using the search field (that has the debounce feature) instead of the text box.

Downside of this is along the lines of what I mentioned above to @Joyce —

• Forced to one line (example), which makes it inconvenient for long notes
• formatting/alignment is centered, and you can’t change it

[clncsports]

Hi @aurelie . How does this depend? You mean adding a filter at the collection level may NOT actually request a filtered response from the data source?

I’m particularly concerned with Airtable.

If I add a filter within the collection configuration, is the intended behavior that the response will only include the filtered results?

Thanks

EDIT: to elaborate a bit further on what specifically I’m seeking to confirm…

I have a users table in airtable that contains a field for auth0 ID, which is what I want to filter by. Based on the editor UI in weweb, it seems apparent that if I want to make a call to my airtable table and retrieve a record for only my authenticated user, I would simply need to add a filter at the user collection level that filters like “where auth0ID = <auth0 ID from auth0 plugin*”.

In the editor, adding that filter results in that collection only returned 1 record, as expected.

Is this how it is intended to behave in a published app too? That only that 1 record that meets the filter conditions would be returned?

[aurelie]

Hi @clncsports ,
This will work the same way in your published app.
BUT
Be aware that this is not secure. In your app you expose and fetch only the data of the user, but the endpoint you are calling is available for anyone who knows how to call them (with Postman for example). And they can provide any userId they want (or none) and get all your data. So if your data are sensitive, this is not a good solution.
To be secure, the filter by userId must be done on the backend side.

[clncsports]

Thanks, @aurelie this is what I expected.

In this case, I believe I’m seeing a bug with the airtable backend filters in production.

In 2 published apps I’m seeing the same thing. I have a backend filter applied on a users table using the auth0 plugin variable. When I inspect the call in production, I see it returning the full table.

here you can see the filter variable in the request payload:

yet the response includes all 263 records from the table:

here’s the filter on the collection in the editor:

The binded variable is 'Auth0 - [‘user’][‘sub’]

The field in airtable is indeed of type string:

[Alexis]

Hi @clncsports,

If you check the API call in the editor you will notice the same behavior, they are not backend filters but frontend filters, you will always get the complete collection and then the collection will be reduced in the frontend side by filters and pagination.

We are actively working on adding backend filters to Airtable.