February 2017

Issues accessing Gateway and Api
The incident was related to equipment power issues. Most services were restored within 15 minutes. Below is a list of services affected: API/WEB Processing: 10:46 AM thru 11:00 AM CST FTP Processing: 10:46 AM thru 12:12 PM CST Users may also have experienced higher latency with the fraud management platform between 11:00 AM thru 11:45 AM CST.
Feb 16, 10:48-11:30 CST

January 2017

Phone system issues
The phone system issue impacting our ability to answer calls has been resolved.
Jan 27, 11:39-12:29 CST
Control Panel access issues
We saw sporadic connectivity issues to the control panel from starting at 7:11 PM Central time. The connection issues started tapering off at 7:17 PM Central time and saw no more connectivity issues after 7:21 PM Central time. API processing was unaffected by this incident and operated normally.
Jan 2, 19:17-19:43 CST

December 2016

Electronic check processing issues
We have not seen any additional impacted transactions since 5:03 AM Central time.
Dec 10, 05:06-05:43 CST

November 2016

First Data Nashville processing issues
We have not seen any further disruption on the First Data Nashville platform. We saw impacted transactions from 5:11 PM Central time until 5:55 PM Central time.
Nov 30, 17:15-18:12 CST
FDMS Nashville Processing Issues
We saw sporadic impacted transactions with the highest volume between 6:00 PM Central and 6:22 PM Central and have not seen any additional impacted transactions since.
Nov 13, 18:24-19:28 CST
Platform Connectivity Issues
Connectivity to the platform has remained stable. Connectivity issues were related to a confirmed incident with one of our internet carriers. The largest impact was between 5:01 - 5:02 PM CDT and 5:14 - 5:17 PM CDT. We have completely routed off of the carrier until we have confirmation that issues have been resolved.
Nov 11, 17:05-17:54 CST
Electronic check processing issues
We saw sporadic transactions issues with our ACH vendor from 8:15 PM CST to 11:00 PM CST. Our vendor has confirmed that the issue has been resolved.
Nov 8, 20:18-23:55 CST

October 2016

FDMSO processing issues
We have not seen any more problematic transactions on the FDMSO platform. We saw inconsistent impact beginning at 2:00 PM Central time and lasting until 2:16 PM Central time.
Oct 26, 14:17-14:52 CST
DNS Resolution Issues
We have received notice from our DNS provider that service has been restored. The entire scope of impact is very difficult to ascertain. The issues were often isolated to certain regions and intermittent. Furthermore, we implemented mitigation tactics throughout the day which further reduced some of the impact. In all, the DNS vendor is reporting that the bulk of the issues occurred 6:10 am thru 12:45 pm CDT. Several hours ago, we began the process of shifting domains to an alternate provider. We will continue to do so to ensure that any further DoS attacks that may be targeted at our DNS provider have minimal impact on our platform.
Oct 21, 11:21-17:38 CST
Connectivity Issues relating to DNS
No further issues have been observed. The issue was related to a DDoS attack against our DNS provider. The provider identified that the initial impact was at 6:10 AM CDT. They determined that services were completely restored by 8:20 AM CDT. During this time, DNS queries in certain regions (predominately US East Coast) may have taken longer to complete or completely timed out. This may have affected the ability for certain customers to connect to the platform.
Oct 21, 07:13-09:27 CST
Intermittent TSYS Processing Errors
We have observed no further errors on the TSYS platform. In total, we saw timeouts with TSYS processing ranging from approximately 1% to 5% of total authorizations between 5:06 PM and 8:13 PM CDT.
Oct 14, 17:13-21:02 CST
Isolated Carrier Routing Issues
No further routing anomalies have been observed. Isolated connectivity issues were reported between 11:42 am and 12:04 pm CDT. Impact was limited to a region in the southern mid-west, affecting only a subset of customers within that region.
Oct 13, 12:02-12:57 CST
Global Payments Settlement Issues
All affected settlements have been successfully settled. Transactions that received erroneous response data from Global Payments were withheld from the settlement. Partners who were impacted will be notified.
Oct 10, 23:55 - Oct 11, 19:17 CST
Phone system issues
We have not seen any subsequent problems with our phone system since it recovered.
Oct 4, 09:46-11:21 CST

September 2016

First Data Nashville Timeouts
In total, we observed intermittent timeouts to the First Data Nashville from 11:19 thru 11:28 AM CDT. We have observed no further errors.
Sep 27, 11:28-12:08 CST
Gateway connectivity issues
We saw sporadic packet loss with a couple high points from 9:42 PM CDT until 9:44 PM CDT and 9:52 PM CDT until 9:54 PM CDT. We have not seen any additional connectivity problems to the gateway since 9:54 PM.
Sep 24, 21:50-22:41 CST
TSYS processing issues
We have not seen any impact other than the times listed and the issues stemmed from an issue with an upstream ISP.
Sep 22, 00:37-01:24 CST
Phone system issues
We have not seen any further issues on the phone system since service resumed.
Sep 15, 08:15-09:01 CST
Gateway performance issues
We have not seen any other issues or anomalies after 1:28 PM CDT when the issue was resolved.
Sep 13, 13:20-14:10 CST
Phone system issues
We experienced 3 minutes of phone issues from 12:31 CDT until 12:34 CDT and have not seen any further issues with the phone system.
Sep 7, 12:32-13:18 CST

August 2016

Tsys processing issues
We saw sporadic transactions issues on the TSYS platform from 1:03 AM CDT until 1:32 AM CDT and have not seen any additional impact since. We have also received confirmation from TSYS that the problem is resolved.
Aug 21, 01:34-02:04 CST
ACH Processing issues
We have seen no further exceptions or connection issues to our ACH provider. We saw repeated connectivity issues and bouts of slow responses from 4:05 AM until 5:03 AM CDT.
Aug 19, 04:19-05:54 CST
Global Payments processing issues
We have observed no further errors on the Global Payments network. Errors were intermittent and we're observed between 4:50 pm and 5:46 pm CDT
Aug 17, 17:45-18:59 CST
ACH Processing issues
ACH processing issues started at 3:13 AM CDT and continued until 9:05 AM CDT and we have not seen any further problematic transactions since.
Aug 15, 04:44-09:32 CST
Intermittent Carrier Issues
There has been no further latency on any of our upstream carriers.
Aug 3, 23:16-23:46 CST

July 2016

Site accessibility issues
We were able to identify that the intermittent issue from 11:53 AM CDT until 11:55 AM CDT was related to a performance issue caused by a recently released feature within our application. The problematic component was isolated and will be evaluated. There should be no further customer impact related to this component.
Jul 19, 11:56-12:20 CST

June 2016

First Data Omaha processing issues
We have not seen any further impact since 12:32 PM CDT.
Jun 17, 12:32-15:01 CST
Issues with site accessibility
We have not seen any more impact on connectivity since 4:24 PM CDT. This outage was related to a device in our core networking infrastructure. The node that crashed was part of a redundant cluster, however, a failure in the cluster fail-over resulted in a longer than expected impact. Engineers are working with the vendor to isolate the redundancy issue and ensure that node fail-over within a cluster is as seamless as possible.
Jun 13, 16:18-17:21 CST

May 2016

Phone support provider issues
The phones have been stable since service was restored.
May 12, 10:46-12:39 CST

April 2016

Vantiv Processing Issues
We have not seen any further impacted transactions on the Vantiv platform since 2:23 PM CDT and consider the incident to be resolved.
Apr 23, 14:11-14:57 CST
TSYS Processing issues
We saw several but infrequent impacted transactions from 11:00 AM until 11:53 AM CDT on the TSYS platform and it has been error free since.
Apr 16, 12:06-12:57 CST
Paymentech Tampa Processing Issues
We have not seen any additional impacted transactions since 10:04 PM CDT.
Apr 12, 22:04-22:50 CST
First Data Omaha Processing Issues
We have not seen any further issues since the First Data Omaha platform came back up. First Data Omaha reported issues between 4:30 PM CDT and 8:55 PM CDT and we have not seen any impacted transactions since that time.
Apr 10, 18:53-22:23 CST
Vantiv (previously Litle) Processing issues
We saw impacted transactions on the Vantiv (previously Litle) from 1:28 PM CDT until 1:41 PM CDT and have not seen any additional impacted transactions since.
Apr 4, 13:40-14:20 CST

March 2016

Pay.on Processing Issues
We have not seen any additional impacted transactions on the Pay.on platform since 3:28 AM CDT.
Mar 23, 02:24-04:02 CST
Vantiv Processing Issues
The errors on the Vantiv platform were intermittent between 12:53 CST and 2:15 CST and we have not seen any additional errors since.
Mar 10, 13:27-15:10 CST
TSYS Processing Issues
No further errors have been observed on the TSYS platform. Errors were seen from 3:02 AM thru 4:38 AM CST with increasing frequency as time went on peaking at about 25% impact.
Mar 9, 03:46-05:04 CST
Sporadic TSYS Processing Issues
No further issues have been observed.
Mar 6, 22:36-23:09 CST
Trouble accessing API
We have seen no additional slowness since 11:22.
Mar 3, 11:23-12:59 CST

February 2016

Elavon Processing Issues
Elavon had a confirmed partial outage from 2:36 CST until 3:00 PM CST. We have not seen any impacted transactions since that time.
Feb 22, 14:41-16:23 CST
First Data Processing Issues
We have not observed any further errors. The first error on the First Data Nashville/North platforms was observed at 3:36 PM CST and while still relatively sporadic, they became worse around 5:00 pm CST. The last error was observed at 5:17 PM CST.
Feb 21, 17:05-17:56 CST

January 2016

Customer Service Phone Issues
Our telecom vendor has confirmed that the issues have been resolved.
Jan 13, 15:00-15:57 CST
TSYS Processing Issues
In total, TSYS errors we're observed between 10:29am thru 10:35am CST. We are still waiting on TSYS for the full details regarding the incident.
Jan 12, 10:32-11:28 CST
Global Processing Issues
This incident has been resolved.
Jan 4, 09:40-13:08 CST

December 2015

Isolated carrier issues
This incident has been resolved.
Dec 8, 10:08-13:02 CST
Sporadic processing issues on First Data Omaha platform
We have not seen any more issues on the First Data Omaha platform since our last update.
Dec 4, 10:59-12:59 CST
Marketing site access issues
We have not seen any issues on our marketing since it came back up at 12:30 CST and now believe it to be stable.
Dec 3, 12:14-13:10 CST

November 2015

TSYS Processing Issues
We have not observed any further errors. In total, we received intermittent error responses from TSYS between 12:11 PM thru 12:16 PM CST.
Nov 12, 12:14-12:39 CST
Isolated Carrier Issues
As of 7:38 PM CST, carrier routing has remained stable. The incident was sporadic and isolated to a single carrier.
Nov 6, 19:25-20:21 CST
TSYS Processing Issues
There have been no further errors since 6:43 PM CDT. Between 6:15 PM and 6:43 PM CST, approximately 1 for every 20 transactions destined for the TSYS platform timed out waiting for a response from TSYS.
Nov 5, 18:18-19:10 CST

October 2015

ACH Processing Issues
We received confirmation from our vendor that the issue has been resolved. We observed intermittent ACH errors in the minutes leading up to the event and the majority of ACH errors between 7:07 AM and 7:21 AM CDT.
Oct 30, 07:16-07:33 CST
TSYS/MeS Processing Issues
The DNS provider used by TSYS and MeS appears to have returned to a normal state. We will continue to utilize cached IP addresses until we've confirmed their issues have been resolved. In total, the majority of the TSYS/MeS errors occurred between 3:42 PM CDT thru 4:00 PM CDT. We observed sporadic errors on these processors through approximately 4:30 PM CDT.
Oct 15, 15:50-17:38 CST
Paymentech Salem Processing Issues
Paymentech Salem processing has remained stable. Errors were isolated to the Paymentech Salem platform. These errors were observed between 5:52 - 6:12 am CDT.
Oct 13, 06:05-06:30 CST

September 2015

No incidents reported for this month.

August 2015

Intermittent FDMSO Processing Issues
The highest number of connectivity errors occurred in the minutes surrounding 11:50 AM CDT. Errors were sporadic but enough to raise alarm. By 11:55 AM CDT, the sporadic errors dropped even more significantly. We will continue to monitor the situation.
Aug 27, 11:54-12:22 CST
Internet Routing Issues
We continue to monitor the situation but are closing this incident. In summary, we observed carrier issues between 4:28 - 4:31 PM CDT, during which traffic automatically transitioned off the affected carrier. We saw sporadic issues on the Elavon and First Data platforms for a few minutes after 4:31 PM.
Aug 23, 16:31-17:06 CST
Intermittent Connectivity Issues
Connectivity has remained stable since 1:12 PM CDT.
Aug 6, 13:14-13:42 CST

July 2015

Phone Technical Support Issues
This incident has been resolved.
Jul 27, 10:34-11:15 CST

June 2015

FTP Service Issues
During the incident, FTP service was unavailable from 5:53 PM through 6:13 PM CDT.
Jun 14, 18:07-19:15 CST
Transaction API Issues
Customers experienced intermittent connectivity from 3:10 PM thru 4:15 PM CDT. We continued to see routing issues on the Credomatic and Elavon platform through approximately 5:15 PM CDT.
Jun 1, 15:22-19:02 CST

May 2015

Electronic Check Processing Incident
We received confirmation from our partner that the processing incident has been resolved. Isolated errors on the Electronic Check processing platform began to occur at 9:04 AM CDT. The impact continued to grow to until it was resolved at 9:59 AM CDT. Only Electronic Check transactions were impacted.
May 23, 09:29-10:21 CST
Global Processing Issues
No further errors have been observed on the Global Payments platform as of 3:10 pm CDT.
May 20, 14:14-15:43 CST
Electronic Check Processing Issues
We have not observed any errors since 5:30 PM CDT. We've identified that only a small number of transaction were affected. The errors ocurred between 5:01 PM thru 5:30 PM CDT.
May 15, 17:20-17:54 CST
Intermittent Phone Support Issues
We've received confirmation from our vendor that all issues have been resolved. They identified and resolved issues with certain carrier networks between 11:00 AM and 3:00 PM CDT.
May 13, 11:26-17:18 CST
Third-Party ACH Processing Issues
No further errors have been observed and the Third-Party ACH Processor has confirmed that issues are resolved. Sporadic errors were observed as early as 9:46 PM, however, the frequency of errors increased after 10:00 PM CDT. In total, the incident resulted in intermittent Electronic Check Processing errors on the GETI platform from 9:46 PM thru 11:00 PM CDT.
May 8, 22:15-23:30 CST

April 2015

Third-Party Processor Connectivity
No further errors have been observed. We've determined that carrier related connectivity issues were limited to First Data, Global and MeS. Connectivity issues were intermittent between 3:26 am thru 3:32 am CDT for these processors.
Apr 25, 03:32-03:54 CST
TSYS Settlement Delays (1:00 am CDT Cut-Off)
We observed settlement issues on the TSYS platform affecting a number settlements scheduled for Apr 22 at 1:00 AM CDT. TSYS confirmed that they experienced issues during this time. These settlements were delayed and settled after we observed sustained stability.
Apr 22, 05:54 CST
Control Panel Issues
We've determined that the Control Panel and Query API experienced intermittent connectivity issues between 9:44pm - 9:47pm CDT. The Control Panel has remained available since 9:47pm CDT. The Transaction API was not affected.
Apr 21, 21:46-22:18 CST
Paymentech Salem Settlement Delays (12:45 am - 2:15 am CDT)
We observed settlement issues on the Paymentech Salem platform between 12:45 am - 2:15 am CDT. Settlements queued during this time were delayed and settled after we observed sustained stability in the early morning. We also observed sporadic Paymentech authorization errors between 12:44 am thru 12:49 am CDT.
Apr 14, 02:39 CST
First Data Nashville Processing Issues
We observed errors on the First Data Nashville processing network between 12:31 AM thru 12:34 AM CDT. There have been no errors identified since 12:34 AM CDT.
Apr 7, 00:41 CST
Vantiv Processing Issues
We have not seen an error on the Vantiv platform since 3:44 PM CDT. We are considering the issue resolved. In total, we identified sporadic Vantiv errors between 2:17 PM CDT thru 3:44 PM CDT.
Apr 4, 14:32-15:54 CST
Paymentech Tampa Processing Issues
Paymentech Tampa has remained stable.
Apr 3, 19:26-20:19 CST
First Data Omaha / Nashville Processing Issues
No further issues have been identified. Intermittent First Data Omaha / Nashville issues were observed between 8:17 PM CDT thru 8:20 PM CDT.
Apr 2, 20:25-21:08 CST

March 2015

Carrier Issues
As of 1:30 AM CST, engineers have not identified any further issues.
Mar 6, 01:27-02:13 CST

February 2015

No incidents reported for this month.