Amazon SQS

Availability:

Airbyte Cloud Airbyte OSS PyAirbyte

Support Level:

Marketplace

Connector Version:

0.1.1(Last updated 4 months ago)

CDK Version:

0.58.2

Sync Success Rate:

Usage Rate:

Overview

The Amazon SQS source syncs the SQS API, refer: https://docs.aws.amazon.com/AWSSimpleQueueService/latest/APIReference/Welcome.html. It supports full refresh sync and could be viewed in builder with low-code support.

Getting started

Requirements

• AWS IAM Access Key
• AWS IAM Secret Key
• AWS SQS Queue URL
• AWS Region
• Action target

Setup guide

• Create IAM Keys
• Create SQS Queue

Supported Streams

This Source is capable of syncing the following core Action that would be recieved as streams for sync:

• RecieveMessage
• QueueAttributes

Other Actions are in beta which might require more/less parameters

Features

Feature	Supported?(Yes/No)	Notes
Full Refresh Sync	Yes
Incremental Sync	No
Namespaces	No

Performance considerations

The rate of lookup requests for RecieveMessage stream is limited to two per second, per account, per region. This connector gracefully retries when encountering a throttling error. However if the errors continue repeatedly after multiple retries (for example if you setup many instances of this connector using the same account and region), the connector sync will fail.

Output schema

This source will output one stream for the configured SQS Queue. The stream record data will have three fields:

• id (a UUIDv4 as a STRING)
• body (message body as a STRING)
• attributes (attributes of the messages as an OBJECT or NULL)

Properties

Required properties are 'Queue URL', 'AWS Region' and 'Delete Messages After Read' as noted in bold below.

• Queue URL (STRING)
  • The full AWS endpoint URL of the queue e.g. https://sqs.eu-west-1.amazonaws.com/1234567890/example-queue-url
• AWS Region (STRING)
  • The region code for the SQS Queue e.g. eu-west-1
• Max Batch Size (INTEGER)
  • The max amount of messages to consume in a single poll e.g. 5
  • Minimum of 1, maximum of 10
  • Default: 10
• Max Wait Time (INTEGER)
  • The max amount of time (in seconds) to poll for messages before commiting a batch (or timing out) unless we fill a batch (as per Max Batch Size)
  • Minimum of 1, maximum of 20
  • Default: 20
• Message Attributes To Return (STRING)
  • A comma separated list of Attributes to return for each message
  • Default: All
• Message Visibility Timeout (INTEGER)
  • After a message is read, how much time (in seconds) should the message be hidden from other consumers
  • After this timeout, the message is not deleted and can be re-read
  • Default: 20
• AWS IAM Access Key ID (STRING)
  • The Access Key for the IAM User with permissions on this Queue
• AWS IAM Secret Key (STRING)
  • The Secret Key for the IAM User with permissions on this Queue
• Target (STRING)
  • The targeted action resource for the fetch

Reference

Config fields reference

Field

Type

Property name

string

queue_url

string

region

boolean

delete_messages

integer

max_batch_size

integer

max_wait_time

string

attributes_to_return

integer

visibility_timeout

string

access_key

string

secret_key

Changelog

Expand to review

Version	Date	Pull Request	Subject
1.0.0	2024-11-07	41064	Migrate to low code
0.1.1	2024-01-03	#33924	Add new ap-southeast-3 AWS region
0.1.0	2021-10-10	#0000	Initial version