Hey team! Looking at the open connector PRs (Slack, OneDrive, Dropbox, file upload), there's a lot of repeated BaseConnector plumbing across them. I drafted what a @archestra/connector-sdk package + scaffold could look like to cut that. Open to an RFC issue, or do you prefer keeping connectors in-tree?
Hi @user
Pretty cool idea, @user or @user are the authors of this integration. We need to check with them
joey (archestra team) —
hey 👋 I'm not sure we need an external package for this. Do you have specific examples of "repeated BaseConnector plumbing"? If there's particular repetition you notice I think extracting that into `BaseConnector` and/or shared utils would be fine 🙂
André Ahlert —
Hey @user tks!
Fair, in-tree first makes sense while the interface still moves.
Context on where I see this going: short term extract the repeated bits into BaseConnector + utils. Medium term, once the Connector / SyncBatch / checkpoint shapes stabilize, that same code lives behind a public archestra/connector-sdk + create-archestra-connector scaffold (Backstage / Airflow providers / ESLint plugins style).
André Ahlert —
In Long term that opens a connector marketplace, third parties ship zendesk / hubspot / freshdesk without forking core, and Archestra captures distribution + revenue share on paid ones. So I'd treat the in-tree extraction as step 0 of that path, not a separate thing.
André Ahlert —
what I hit reading dropbox + onedrive + slack #3962 was
joey (archestra team) —
> `archestra/connector-sdk` + third parties ship zendesk / hubspot / freshdesk without forking core
how would that work? at the end of the day, the code for the available connectors need to be in the codebase/docker-images themselves. what _concrete_ problem is this trying to solve that exists _today_?
do you have any examples of "repeated bits"?
André Ahlert —
let me get here
André Ahlert —
sorry, came to my pc was on mobile
André Ahlert —
repetition I see in-tree:
André Ahlert —
validateConfig is the same skeleton in dropbox / onedrive / notion / jira / github (parseX + null check + return). slack diverged with per-field errors, which is itself a sign the simple skeleton hit its ceiling.
for example
dropbox: github.com/archestra-ai/archestra/blob/main/platform/backend/src/knowledge-base/connectors/dropbox/dropbox-connector.ts#L56-L64
imho could be BaseConnector.validateConfigWithSchema(schema, label).
André Ahlert —
testConnection has the same try/catch + log + return shape, but error formatting already drifts. dropbox, onedrive, notion use extractErrorMessage. Jira and github do error instanceof Error ? .message : String(error) inline.
and I think jira: github.com/archestra-ai/archestra/blob/main/platform/backend/src/knowledge-base/connectors/jira/jira-connector.ts#L65-L95
André Ahlert —
we could have something like BaseConnector.runConnectionTest({ label, probe })
André Ahlert —
I didnt dig all of then
André Ahlert —
just some notes, came at this from a community product angle (lower the bar for outside contributors, faster connector PR throughput) and ended up zooming in on where the real friction is, which is the boilerplate every new connector reintroduces
André Ahlert —
Parking the SDK/marketplace bit, that was me jumping ahead. What I see as _Concrete_ today (imho):
ValidateConfig is the same skeleton in dropbox / onedrive / notion / jira / github (parse + null check + return).
TestConnection has the same try/catch shape but error formatting already drifts (extractErrorMessage in dropbox/onedrive/notion vs inline instanceof Error in jira/github). Slack diverged into per-field errors, which is the skeleton hitting its ceiling.
Proposing a small PR: BaseConnector.validateConfigWithSchema(schema, label) + BaseConnector.runConnectionTest({ label, probe }), migrating dropbox/onedrive/notion as proof. If it holds across the next 2-3 connectors, we can revisit whether the surface deserves a stable package later. wdyt?
Hey team! Looking at the open connector PRs (Slack, OneDrive, Dropbox, file upload), there's a lot of repeated BaseConnector plumbing across them. I drafted what a @archestra/connector-sdk package + scaffold could look like to cut that. Open to an RFC issue, or do you prefer keeping connectors in-tree?
Hey team! Looking at the open connector PRs (Slack, OneDrive, Dropbox, file upload), there's a lot of repeated BaseConnector plumbing across them. I drafted what a @archestra/connector-sdk package + scaffold could look like to cut that. Open to an RFC issue, or do you prefer keeping connectors in-tree?
19 replies
Ildar Iskhakov (archestra team)10:28 PMOpen in Slack
Hi @user
Pretty cool idea, @user or @user are the authors of this integration. We need to check with them
hey 👋 I'm not sure we need an external package for this. Do you have specific examples of "repeated BaseConnector plumbing"? If there's particular repetition you notice I think extracting that into BaseConnector and/or shared utils would be fine 🙂
Fair, in-tree first makes sense while the interface still moves.
Context on where I see this going: short term extract the repeated bits into BaseConnector + utils. Medium term, once the Connector / SyncBatch / checkpoint shapes stabilize, that same code lives behind a public archestra/connector-sdk + create-archestra-connector scaffold (Backstage / Airflow providers / ESLint plugins style).
In Long term that opens a connector marketplace, third parties ship zendesk / hubspot / freshdesk without forking core, and Archestra captures distribution + revenue share on paid ones. So I'd treat the in-tree extraction as step 0 of that path, not a separate thing.
archestra/connector-sdk + third parties ship zendesk / hubspot / freshdesk without forking core
how would that work? at the end of the day, the code for the available connectors need to be in the codebase/docker-images themselves. what concrete problem is this trying to solve that exists today?
validateConfig is the same skeleton in dropbox / onedrive / notion / jira / github (parseX + null check + return). slack diverged with per-field errors, which is itself a sign the simple skeleton hit its ceiling.
testConnection has the same try/catch + log + return shape, but error formatting already drifts. dropbox, onedrive, notion use extractErrorMessage. Jira and github do error instanceof Error ? .message : String(error) inline.
just some notes, came at this from a community product angle (lower the bar for outside contributors, faster connector PR throughput) and ended up zooming in on where the real friction is, which is the boilerplate every new connector reintroduces
Parking the SDK/marketplace bit, that was me jumping ahead. What I see as Concrete today (imho):
ValidateConfig is the same skeleton in dropbox / onedrive / notion / jira / github (parse + null check + return).
TestConnection has the same try/catch shape but error formatting already drifts (extractErrorMessage in dropbox/onedrive/notion vs inline instanceof Error in jira/github). Slack diverged into per-field errors, which is the skeleton hitting its ceiling.
Proposing a small PR: BaseConnector.validateConfigWithSchema(schema, label) + BaseConnector.runConnectionTest({ label, probe }), migrating dropbox/onedrive/notion as proof. If it holds across the next 2-3 connectors, we can revisit whether the surface deserves a stable package later. wdyt?
