Wednesday, December 28, 2016

Dynamodb restore using datapipeline with newer emr version

11 comments
We faced an interesting issue here.

We had a task to restore dynamodb from backup stored in s3. We took the backup using AWS pipeline. But when we started restoring, which we had been doing for a long time, we found the cluster was not able to provisioned and it was failing on bootstraping.
Digging more into it we found the AWS Dynamodb import template of datapipeline use default subnet under default VPC. Now, we don't have Internet Gateway for this VPC. Since, EMR needs IG to work successfully, this pipeline was failing with "internal error"

We started talking with AWS support. After some discussion, we changed our private subnet. Again the EMR provisioning was failing as the template to restore dynamodb that provided by AWS uses AMI Version 3.9.0 which does not support private subnet.

So we decided to change the AMI Version 3.9 to release label "emr-4.5.0" which we have been using for all our EMR so far. Again we failed with error:
Unable to create resource for @EmrClusterForLoad_2016-12-28T20:33:21 due to: The supplied bootstrap action(s): 'bootstrap-action.7237c1e1-31de-4c02-ae68-c546dd581732' are not supported by release 'emr-4.5.0'. (Service: AmazonElasticMapReduce; Status Code: 400; Error Code: ValidationException; Request ID: e8be350e-cd3c-11e6-8e60-cb10b4c3228c)

That is, the template script provided by AWS does not support emr release label 4.5.0. To overcome the problem, we had to modify EmrCluster bootstrap action in pipeline definition which was:
s3://#{myDDBRegion}.elasticmapreduce/bootstrap-actions/configure-hadoop, --mapred-key-value,mapreduce.map.speculative=false

This was only supported by AMI 3.9.0. For release label emr-4.5.0, it should be added as configuration properties as follows:
--
        {
            "configuration": {
                "ref": "EmrConfigurationId_XXWNE"
            },
            "releaseLabel": "emr-4.5.0",
            "type": "EmrCluster",
            ...
       },
       {
            "property": {
                "ref": "PropertyId_3ghq7"
            },
            "type": "EmrConfiguration",
            "id": "EmrConfigurationId_XXWNE",
            "classification": "mapred-site",
            "name": "DefaultEmrConfiguration1"
        },
        {
            "key": "mapreduce.map.speculative",
            "type": "Property",
            "id": "PropertyId_3ghq7",
            "value": "false",
            "name": "DefaultProperty1"
        },
--

Now, we exported the pipeline definition and added the above configuration. The final pipeline definition was looking like this:

{
  "objects": [
    {
      "property": [
        {
          "ref": "PropertyId_3ghq7"
        }
      ],
      "name": "DefaultEmrConfiguration1",
      "id": "EmrConfigurationId_XXWNE",
      "type": "EmrConfiguration",
      "classification": "mapred-site"
    },
    {
      "output": {
        "ref": "DDBDestinationTable"
      },
      "input": {
        "ref": "S3InputDataNode"
      },
      "maximumRetries": "1",
      "name": "TableLoadActivity",
      "step": "s3://dynamodb-emr-#{myDDBRegion}/emr-ddb-storage-handler/2.1.0/emr-ddb-2.1.0.jar,org.apache.hadoop.dynamodb.tools.DynamoDbImport,#{input.directoryPath},#{output.tableName},#{output.writeThroughputPercent}",
      "runsOn": {
        "ref": "EmrClusterForLoad"
      },
      "id": "TableLoadActivity",
      "type": "EmrActivity",
      "resizeClusterBeforeRunning": "true"
    },
    {
      "subnetId": "subnet-xxxxxxx",
      "name": "EmrClusterForLoad",
      "coreInstanceCount": "1",
      "coreInstanceType": "m3.xlarge",
      "releaseLabel": "emr-4.5.0",
      "id": "EmrClusterForLoad",
      "masterInstanceType": "m3.xlarge",
      "region": "#{myDDBRegion}",
      "type": "EmrCluster",
      "terminateAfter": "23 Hours",
      "configuration": {
                "ref": "EmrConfigurationId_XXWNE"
            }
    },
    {
      "failureAndRerunMode": "CASCADE",
      "resourceRole": "PSS-BDP-QA-DataPipelineDefaultResourceRole",
      "pipelineLogUri": "s3://pss-bdp-qa-logfiles/datapipeline-logs/PSS-BDP-SQA-Dynamodb-Import-1/",
      "role": "PSS-BDP-DataPipelineDefaultRole",
      "scheduleType": "ONDEMAND",
      "name": "Default",
      "id": "Default"
    },
    {
      "writeThroughputPercent": "#{myDDBWriteThroughputRatio}",
      "name": "DDBDestinationTable",
      "id": "DDBDestinationTable",
      "type": "DynamoDBDataNode",
      "tableName": "#{myDDBTableName}"
    },
    {
      "directoryPath": "#{myInputS3Loc}",
      "name": "S3InputDataNode",
      "id": "S3InputDataNode",
      "type": "S3DataNode"
    },
    {
        "key": "mapreduce.map.speculative",
        "type": "Property",
        "id": "PropertyId_3ghq7",
        "value": "false",
        "name": "DefaultProperty1"
    }
  ],
  "parameters": [
    {
      "description": "Input S3 folder",
      "id": "myInputS3Loc",
      "type": "AWS::S3::ObjectKey"
    },
    {
      "description": "Target DynamoDB table name",
      "id": "myDDBTableName",
      "type": "String"
    },
    {
      "default": "0.25",
      "watermark": "Enter value between 0.1-1.0",
      "description": "DynamoDB write throughput ratio",
      "id": "myDDBWriteThroughputRatio",
      "type": "Double"
    },
    {
      "default": "us-east-1",
      "watermark": "us-east-1",
      "description": "Region of the DynamoDB table",
      "id": "myDDBRegion",
      "type": "String"
    }
  ],
  "values": {
    "myDDBRegion": "us-east-1",
    "myDDBTableName": "TABLE_TEST",
    "myDDBWriteThroughputRatio": "1",
    "myInputS3Loc": "s3://my-dynamobackup/TABLE_TEST_201609/2016-12-22-22-55-57"
  }
}

11 Responses so far

  1. Escorts in Mumbai provides Independent Mumbai Escorts and stylish high class Mumbai.

  2. If you want Escort Service in Mumbai Then Mumbai Escorts is the good website for Mumbai Escorts Services.

  3. rosemary says:

    I Saw Your Website Carefully. Thanks For Create Such Kind Of Informative Website. Your All Content Is Relevant To Your Subject. I Say Keep It Continue, Because Your Website And Subject Is Meaningful For The Users. I Have Recommended Your Website With My Friends Also.
    I Am Ritu Sharma. I Am An Independent Mumbai Escort Girl. I Deal In Mumbai Escorts. I Am Very Hot And Sexy. Being A Mumbai Escort Girl I Have Five Years’ Experience. That’s Why I Understand The Each Personal Needs Of My Every Client. My Service Charge Is Low And Service Is Super. You Can Avail My VIP Mumbai Escorts Service At Your Home Or In Hotel. I Am Comfortable To Provide Mumbai Escorts Service In Local And Outside Also. Visit www.escortservicemumbai.net Call +91- 9OO4OO9481
    Visit The Links Below And Have A Look At My Various Mumbai Escort Services-
    Mumbai Escorts
    Mumbai Call Girls
    Mumbai Escorts In Vile
    Parle

    Bandra Escorts
    Juhu Escorts
    Andheri Escorts

    Mumbai Escorts In Colaba

  4. rosemary says:


    I Saw Your Website Carefully. Thanks For Create Such Kind Of Informative Website. Your All Content Is Relevant To Your Subject. I Say Keep It Continue, Because Your Website And Subject Is Meaningful For The Users. I Have Recommended Your Website With My Friends Also.
    I Am Ritu Sharma. I Am An Independent Mumbai Escort Girl. I Deal In Mumbai Escorts. I Am Very Hot And Sexy. Being A Mumbai Escort Girl I Have Five Years’ Experience. That’s Why I Understand The Each Personal Needs Of My Every Client. My Service Charge Is Low And Service Is Super. You Can Avail My VIP Mumbai Escorts Service At Your Home Or In Hotel. I Am Comfortable To Provide Mumbai Escorts Service In Local And Outside Also. Visit www.escortservicemumbai.net Call +91- 9OO4OO9481
    Visit The Links Below And Have A Look At My Various Mumbai Escort Services-
    Mumbai Escorts
    Mumbai Call Girls
    Mumbai Escorts In Vile
    Parle

    Bandra Escorts
    Juhu Escorts
    Andheri Escorts

    Mumbai Escorts In Colaba

  5. Thanks For Create Such Kind Of Informative Website. It Is Really Useful According Your Current Topic. Please Keep This Continue To Help The People Who Are Looking For This Topic. After See Your Niche I Have Recommended Your Website With My Friends Also.
    My name is Neha Gupta. I run my independent Mumbai Escorts Service. I am an independent Mumbai escort girl. My service charge is low according the market price. My service is super, because I know very well the personal need of my each customer. I am comfortable to provide my Mumbai Escorts Service at your home or in Hotel. 24X7 I am available. Visit- http://www.nehagupta.biz/ Call- +91 9OO4458359

    Visit The Links Below And Have A Look At My Various Mumbai Escort Services-

    Mumbai Escorts
    Mumbai Call Girls
    Escort Service In Mumbai
    Mumbai Escort Girls
    Bandra Escorts
    Juhu Escorts
    Andheri Escorts

  6. simran says:

    Thanks For Create Such Kind Of Informative Website. All Content Is Relevant To Your Subject. Keep It Continue, Because Your Website And Subject Is Meaningful For The Users. After See Your Niche I Have Recommended Your Website With My Friends Also.
    My Name Is Priya Singh. I Run My Own Mumbai Escorts Service. I Am An Independent Mumbai Escort Girl. I Am Beautiful And Hot. My Service Charge Is Low And Service Is Super. Being Professional I Have Seven Years’ Experience As An Escort Girl. So I Understand And Feel The Real Needs And Requirement Of My Each Client. You Can Taste Me Any Time. According The Convenience You Can Avail My VIP Escort Service At Your Home Or In Hotel Also. To Book My VIP Mumbai Escort Service Call +91 9987215552. Visit http://www.escortagencyinmumbai.com/


    Mumbai Call Girls
    Mumbai Escorts In Vile Parle
    Bandra Escorts
    Juhu Escorts
    Andheri Escorts
    Escort Service In Colaba

  7. My name is Arpita Jain. I run my independent Mumbai Escorts Service. I am an independent Mumbai escort girl. My service charge is low according the current market price. My service is super, because I know very well the personal requirement of my each customer. I am comfortable to provide my Mumbai Escorts Service at your home or in Hotel. Visit- http://www.arpitajain.org/ Call- +91 9OO4458359
    Visit The Links Below And Have A Look At My Various Mumbai Escort Services-

    Mumbai Escorts Service
    Mumbai Escort Girls
    Juhu Escorts

    Escort Service In Mumbai


  8. Unknown says:

    Buy your products online at low rates in Auckland Newzealand.We Provide you quality household items, outdoor furniture nz, gazeboz nz, items and all the items at low rates.

  9. This comment has been removed by the author.

Leave a Reply

Labels