Hey team! Looking at the open connector PRs (Slack, OneDrive, Dropbox, file upload), there's a lot of repeated BaseConnector plumbing across them. I drafted what a @archestra/connector-sdk package + scaffold could look like to cut that. Open to an RFC issue, or do you prefer keeping connectors in-tree?
Hi @user
Pretty cool idea, @user or @user are the authors of this integration. We need to check with them
joey (archestra team) —
hey 👋 I'm not sure we need an external package for this. Do you have specific examples of "repeated BaseConnector plumbing"? If there's particular repetition you notice I think extracting that into `BaseConnector` and/or shared utils would be fine 🙂
André Ahlert —
Hey @user tks!
Fair, in-tree first makes sense while the interface still moves.
Context on where I see this going: short term extract the repeated bits into BaseConnector + utils. Medium term, once the Connector / SyncBatch / checkpoint shapes stabilize, that same code lives behind a public archestra/connector-sdk + create-archestra-connector scaffold (Backstage / Airflow providers / ESLint plugins style).
André Ahlert —
In Long term that opens a connector marketplace, third parties ship zendesk / hubspot / freshdesk without forking core, and Archestra captures distribution + revenue share on paid ones. So I'd treat the in-tree extraction as step 0 of that path, not a separate thing.
André Ahlert —
what I hit reading dropbox + onedrive + slack #3962 was
joey (archestra team) —
> `archestra/connector-sdk` + third parties ship zendesk / hubspot / freshdesk without forking core
how would that work? at the end of the day, the code for the available connectors need to be in the codebase/docker-images themselves. what _concrete_ problem is this trying to solve that exists _today_?
do you have any examples of "repeated bits"?
André Ahlert —
let me get here
André Ahlert —
sorry, came to my pc was on mobile
André Ahlert —
repetition I see in-tree:
André Ahlert —
validateConfig is the same skeleton in dropbox / onedrive / notion / jira / github (parseX + null check + return). slack diverged with per-field errors, which is itself a sign the simple skeleton hit its ceiling.
for example
dropbox: github.com/archestra-ai/archestra/blob/main/platform/backend/src/knowledge-base/connectors/dropbox/dropbox-connector.ts#L56-L64
imho could be BaseConnector.validateConfigWithSchema(schema, label).
André Ahlert —
testConnection has the same try/catch + log + return shape, but error formatting already drifts. dropbox, onedrive, notion use extractErrorMessage. Jira and github do error instanceof Error ? .message : String(error) inline.
and I think jira: github.com/archestra-ai/archestra/blob/main/platform/backend/src/knowledge-base/connectors/jira/jira-connector.ts#L65-L95
André Ahlert —
we could have something like BaseConnector.runConnectionTest({ label, probe })
André Ahlert —
I didnt dig all of then
André Ahlert —
just some notes, came at this from a community product angle (lower the bar for outside contributors, faster connector PR throughput) and ended up zooming in on where the real friction is, which is the boilerplate every new connector reintroduces
André Ahlert —
Parking the SDK/marketplace bit, that was me jumping ahead. What I see as _Concrete_ today (imho):
ValidateConfig is the same skeleton in dropbox / onedrive / notion / jira / github (parse + null check + return).
TestConnection has the same try/catch shape but error formatting already drifts (extractErrorMessage in dropbox/onedrive/notion vs inline instanceof Error in jira/github). Slack diverged into per-field errors, which is the skeleton hitting its ceiling.
Proposing a small PR: BaseConnector.validateConfigWithSchema(schema, label) + BaseConnector.runConnectionTest({ label, probe }), migrating dropbox/onedrive/notion as proof. If it holds across the next 2-3 connectors, we can revisit whether the surface deserves a stable package later. wdyt?
joey (archestra team) —
I think those sound like great cleanup suggestions!
joey (archestra team) —
yeah I mean we can still brainstorm the sdk/marketplace portion, totally ok 🙂 personally I'm just not sure I grok the vision/benefits (my first impression is it _maybe_ is a bit of overengineering?) - but maybe I am missing something in your vision.
maintaining an npm package would be for external codebases to consume?
André Ahlert —
Thanks!
André Ahlert —
yeah, npm package for outside repos to consume. one pushback though: distribution and SDK are separate axes.
Innokentii Konstantinov (archestra team) —
Hi, thanks for the proposal! Why do we need outside repos to consume this package?
André Ahlert —
backstage ships plugins bundled into the final app at build time and still maintains @backstage/* on npm. the package is just like an API contract + types + scaffold,, not a plugin loader. "connectors live in the docker image" doesn't kill the SDK case
André Ahlert —
hey! sorry, i was in the middle of writing hehehe
André Ahlert —
@user external consumption is the eventual shape, not a today-need, for sure
André Ahlert —
an internal workspace package already captures most of the value (typed contract, scaffold, semver discipline)
André Ahlert —
npm publish only mattering if you'd want community contributors shipping niche connectors without going through the core PR cycle
André Ahlert —
customers writing private connectors for internal systems they don't plan to upstream, or splitting archestra's own connectors across repos for independent release cadence.
André Ahlert —
for exemple: community contributors who'd ship niche connectors but don't want to clone and build the whole platform to do it
André Ahlert —
worth noting workspace-vs-npm is a false split. imho monorepo workspaces can publish to npm, same package serves in-tree connectors + external consumers, marginal cost is one CI step. backstage runs exactly that setup, plugins still get bundled into the final app at build time
André Ahlert —
_*so the path I'd actually propose:*_ BaseConnector cleanup PR first, extract the workspace package as the shape stabilizes, npm publish flips on whenever external authoring becomes desirable. the publish step is reversible and cheap, the API surface is the part that (i think 🙂 ) needs care.
Innokentii Konstantinov (archestra team) —
It's an interesting proposal, but I don't see any need in this. Anyone who wants to have a private connector could just maintain their own work. No separate packages needed, it's just an extra-complexity. If you see any potential improvements in BaseConnector structure as a part of existing codebase - we will be happy to review it.
André Ahlert —
got it, thanks. parking the SDK/external thread, if a contributor pattern actually shows up in the wild later it'd be backed by evidence rather than speculation, until then it's not worth the extra surface (sorry guys for the blogpost in here)
André Ahlert —
opening the BaseConnector cleanup PR (validateConfigWithSchema + runConnectionTest, migrating dropbox/onedrive/notion as proof). will tag y'all when it's up
André Ahlert —
PR up, github.com/archestra-ai/archestra/pull/4374. lmk if you want me to split it differently.
Hey team! Looking at the open connector PRs (Slack, OneDrive, Dropbox, file upload), there's a lot of repeated BaseConnector plumbing across them. I drafted what a @archestra/connector-sdk package + scaffold could look like to cut that. Open to an RFC issue, or do you prefer keeping connectors in-tree?
Hey team! Looking at the open connector PRs (Slack, OneDrive, Dropbox, file upload), there's a lot of repeated BaseConnector plumbing across them. I drafted what a @archestra/connector-sdk package + scaffold could look like to cut that. Open to an RFC issue, or do you prefer keeping connectors in-tree?
37 replies
Ildar Iskhakov (archestra team)10:28 PMOpen in Slack
Hi @user
Pretty cool idea, @user or @user are the authors of this integration. We need to check with them
hey 👋 I'm not sure we need an external package for this. Do you have specific examples of "repeated BaseConnector plumbing"? If there's particular repetition you notice I think extracting that into BaseConnector and/or shared utils would be fine 🙂
Fair, in-tree first makes sense while the interface still moves.
Context on where I see this going: short term extract the repeated bits into BaseConnector + utils. Medium term, once the Connector / SyncBatch / checkpoint shapes stabilize, that same code lives behind a public archestra/connector-sdk + create-archestra-connector scaffold (Backstage / Airflow providers / ESLint plugins style).
In Long term that opens a connector marketplace, third parties ship zendesk / hubspot / freshdesk without forking core, and Archestra captures distribution + revenue share on paid ones. So I'd treat the in-tree extraction as step 0 of that path, not a separate thing.
archestra/connector-sdk + third parties ship zendesk / hubspot / freshdesk without forking core
how would that work? at the end of the day, the code for the available connectors need to be in the codebase/docker-images themselves. what concrete problem is this trying to solve that exists today?
validateConfig is the same skeleton in dropbox / onedrive / notion / jira / github (parseX + null check + return). slack diverged with per-field errors, which is itself a sign the simple skeleton hit its ceiling.
testConnection has the same try/catch + log + return shape, but error formatting already drifts. dropbox, onedrive, notion use extractErrorMessage. Jira and github do error instanceof Error ? .message : String(error) inline.
just some notes, came at this from a community product angle (lower the bar for outside contributors, faster connector PR throughput) and ended up zooming in on where the real friction is, which is the boilerplate every new connector reintroduces
Parking the SDK/marketplace bit, that was me jumping ahead. What I see as Concrete today (imho):
ValidateConfig is the same skeleton in dropbox / onedrive / notion / jira / github (parse + null check + return).
TestConnection has the same try/catch shape but error formatting already drifts (extractErrorMessage in dropbox/onedrive/notion vs inline instanceof Error in jira/github). Slack diverged into per-field errors, which is the skeleton hitting its ceiling.
Proposing a small PR: BaseConnector.validateConfigWithSchema(schema, label) + BaseConnector.runConnectionTest({ label, probe }), migrating dropbox/onedrive/notion as proof. If it holds across the next 2-3 connectors, we can revisit whether the surface deserves a stable package later. wdyt?
yeah I mean we can still brainstorm the sdk/marketplace portion, totally ok 🙂 personally I'm just not sure I grok the vision/benefits (my first impression is it maybe is a bit of overengineering?) - but maybe I am missing something in your vision.
maintaining an npm package would be for external codebases to consume?
backstage ships plugins bundled into the final app at build time and still maintains @backstage/* on npm. the package is just like an API contract + types + scaffold,, not a plugin loader. "connectors live in the docker image" doesn't kill the SDK case
customers writing private connectors for internal systems they don't plan to upstream, or splitting archestra's own connectors across repos for independent release cadence.
worth noting workspace-vs-npm is a false split. imho monorepo workspaces can publish to npm, same package serves in-tree connectors + external consumers, marginal cost is one CI step. backstage runs exactly that setup, plugins still get bundled into the final app at build time
so the path I'd actually propose: BaseConnector cleanup PR first, extract the workspace package as the shape stabilizes, npm publish flips on whenever external authoring becomes desirable. the publish step is reversible and cheap, the API surface is the part that (i think 🙂 ) needs care.
Innokentii Konstantinov (archestra team)9:05 AMOpen in Slack
It's an interesting proposal, but I don't see any need in this. Anyone who wants to have a private connector could just maintain their own work. No separate packages needed, it's just an extra-complexity. If you see any potential improvements in BaseConnector structure as a part of existing codebase - we will be happy to review it.
got it, thanks. parking the SDK/external thread, if a contributor pattern actually shows up in the wild later it'd be backed by evidence rather than speculation, until then it's not worth the extra surface (sorry guys for the blogpost in here)
opening the BaseConnector cleanup PR (validateConfigWithSchema + runConnectionTest, migrating dropbox/onedrive/notion as proof). will tag y'all when it's up
