Last updated 1 month ago

Architecture

The diagram below gives a high-level overview of the structure of the Skill Engine API and the systems it interfaces with:

Architecture of the Skill Engine system.

Network Isolation

The Skill Engine has a layered design, deployed within a virtual private cloud. That way, only the Skill Engine API itself can be interfaced with from the outside world, and even this communication can be tightly regulated with measures such as IP whitelisting. Security measures can be controlled per individual customer.

Rate limits

The Skill Engine has a concurrency limiter that limits the number of requests that are active at any given time to 10 per customer. Excessive requests will result in an error response with status code 429.

Components

Skill Engine Core

Central in the system, you can find the Skill Engine Core - this is the AI brain behind the skills intelligence in the Skill Engine API. The Core is entirely stateless and can therefore be leveraged across tenants, making the system more scaleable and robust. The Skill Engine Core communicates with the Vacancy Data Lake, as well as using the Google Vision API to convert PDF documents into text.

Vacancy Data Lake

The Core communicates with the Vacancy Data Lake, which aggregates and analyses vacancies scraped from public sources. For example, this allows the Engine to learn about skill trends, upcoming functions... Importantly, this data lake is used exclusively for public data and does not get into contact with customer information.

Skill Engine API

The next layer up, still inside the TechWolf VPC, is the Skill Engine API itself. This API is responsible for translating the domain driven language offered by the API into the technical one used for interfacing with the core efficiently. In addition, the Skill Engine API is where customer data lives: it connects to a database which stores the skill profiles for this customer. The only way to interface with the Skill Engine system from the outside is through the Skill Engine API, typically done with connectors that plug into your existing HR and knowledge systems. To offer maximal data control, data is pushed to the Skill Engine API (using triggers inside these systems), rather than being pulled at the Skill Engine's initiative.

Customer Database

Data for different customers is stored with logical data separation. All data is encrypted both at rest (AES-256) and in transit (SSL/TLS), with the database only being accessible from within the TechWolf virtual private cloud.

Connectors

Our API is built on open standards: we use a RESTful API design, described with detail through an OpenAPI specification. Each of these aspects enables you to build a connector between your system and the Skill Engine API with ease.

On top of this well-documented interface, we assist our customers in building connectors that bridge the gap between the Skill Engine and popular HR, knowledge sharing and communication systems. While these are currently in their early access stage, they will be open sourced in the future.

Dependencies

The following services form the backbone of the Skill Engine API:

  • AWS as the underlying cloud platform, primarily with the following services:

    • EC2 for compute instances.
    • RDS for managed databases.
    • S3, Glue, Athena and Sagemaker for the Data Lake.
  • Auth0 as our identity and authentication provider.
  • Google Vision to get the text from documents.

Each of these services only processes information for TechWolf inside the European Union.