EC2 User Data Limit Exceeded

[stevejhall]

When following the demo install shown here: https://github.com/FINRAOS/herd/wiki/demo-install

It appears that the user data exceeds an AWS limit. This is the error I got:
The following resource(s) failed to create: [herdApplicationServer]. . Rollback requested by user.
CREATE_FAILED | AWS::EC2::Instance | herdApplicationServer | User data is limited to 16384 bytes (Service: AmazonEC2; Status Code: 400; Error Code: InvalidParameterValue; Request ID: 3a...)

A bit of Googling seems to indicate that AWS has an upper limit of 16K

By way of troubleshooting I hacked out a big chunk of the UserData, just to see if I could get past this error. That test was successful. However, I do not fully understand the impacts of doing so.

This is what I deleted
"psql -f herd.postgres.0.01.0-to-0.02.0.upgrade.sql\n",
"psql -f herd.postgres.0.02.0-to-0.03.0.upgrade.sql\n",
"psql -f herd.postgres.0.03.0-to-0.04.0.upgrade.sql\n",
"psql -f herd.postgres.0.04.0-to-0.05.0.upgrade.sql\n",
"psql -f herd.postgres.0.05.0-to-0.06.0.upgrade.sql\n",
"psql -f herd.postgres.0.06.0-to-0.07.0.upgrade.sql\n",
"psql -f herd.postgres.0.07.0-to-0.08.0.upgrade.sql\n",
"psql -f herd.postgres.0.08.0-to-0.09.0.upgrade.sql\n",
"psql -f herd.postgres.0.09.0-to-0.10.0.upgrade.sql\n",
"psql -f herd.postgres.0.10.0-to-0.11.0.upgrade.sql\n",
"psql -f herd.postgres.0.11.0-to-0.12.0.upgrade.sql\n",
"psql -f herd.postgres.0.12.0-to-0.13.0.upgrade.sql\n",
"psql -f herd.postgres.0.13.0-to-0.14.0.upgrade.sql\n",
"psql -f herd.postgres.0.14.0-to-0.15.0.upgrade.sql\n",
"psql -f herd.postgres.0.15.0-to-0.16.0.upgrade.sql\n",
"psql -f herd.postgres.0.16.0-to-0.17.0.upgrade.sql\n",
"psql -f herd.postgres.0.17.0-to-0.18.0.upgrade.sql\n",
"psql -f herd.postgres.0.18.0-to-0.19.0.upgrade.sql\n",
"psql -f herd.postgres.0.19.0-to-0.20.0.upgrade.sql\n",
"psql -f herd.postgres.0.20.0-to-0.21.0.upgrade.sql\n",
"psql -f herd.postgres.0.21.0-to-0.22.0.upgrade.sql\n",
"psql -f herd.postgres.0.22.0-to-0.23.0.upgrade.sql\n",
"psql -f herd.postgres.0.23.0-to-0.24.0.upgrade.sql\n",
"psql -f herd.postgres.0.24.0-to-0.25.0.upgrade.sql\n",
"psql -f herd.postgres.0.25.0-to-0.26.0.upgrade.sql\n",
"psql -f herd.postgres.0.26.0-to-0.27.0.upgrade.sql\n",
"psql -f herd.postgres.0.27.0-to-0.28.0.upgrade.sql\n",
"psql -f herd.postgres.0.28.0-to-0.29.0.upgrade.sql\n",
"psql -f herd.postgres.0.29.0-to-0.30.0.upgrade.sql\n",
"psql -f herd.postgres.0.30.0-to-0.31.0.upgrade.sql\n",
"psql -f herd.postgres.0.31.0-to-0.32.0.upgrade.sql\n",
"psql -f herd.postgres.0.32.0-to-0.33.0.upgrade.sql\n",
"psql -f herd.postgres.0.33.0-to-0.34.0.upgrade.sql\n",
"psql -f herd.postgres.0.34.0-to-0.35.0.upgrade.sql\n",
"psql -f herd.postgres.0.35.0-to-0.36.0.upgrade.sql\n",
"psql -f herd.postgres.0.36.0-to-0.37.0.upgrade.sql\n",
"psql -f herd.postgres.0.37.0-to-0.38.0.upgrade.sql\n",
"psql -f herd.postgres.0.38.0-to-0.39.0.upgrade.sql\n",
"psql -f herd.postgres.0.39.0-to-0.40.0.upgrade.sql\n",
"psql -f herd.postgres.0.40.0-to-0.41.0.upgrade.sql\n",
"psql -f herd.postgres.0.41.0-to-0.42.0.upgrade.sql\n",
"psql -f herd.postgres.0.42.0-to-0.43.0.upgrade.sql\n",
"psql -f herd.postgres.0.43.0-to-0.44.0.upgrade.sql\n",
"psql -f herd.postgres.0.44.0-to-0.45.0.upgrade.sql\n",
"psql -f herd.postgres.0.45.0-to-0.46.0.upgrade.sql\n",
"psql -f herd.postgres.0.46.0-to-0.47.0.upgrade.sql\n",
"psql -f herd.postgres.0.47.0-to-0.48.0.upgrade.sql\n",
"psql -f herd.postgres.0.48.0-to-0.49.0.upgrade.sql\n",
"psql -f herd.postgres.0.49.0-to-0.50.0.upgrade.sql\n",
"psql -f herd.postgres.0.50.0-to-0.51.0.upgrade.sql\n",
"psql -f herd.postgres.0.51.0-to-0.52.0.upgrade.sql\n",
"psql -f herd.postgres.0.52.0-to-0.53.0.upgrade.sql\n",
"psql -f herd.postgres.0.53.0-to-0.54.0.upgrade.sql\n",
"psql -f herd.postgres.0.54.0-to-0.55.0.upgrade.sql\n",
"psql -f herd.postgres.0.55.0-to-0.56.0.upgrade.sql\n",
"psql -f herd.postgres.0.56.0-to-0.57.0.upgrade.sql\n",
"psql -f herd.postgres.0.57.0-to-0.58.0.upgrade.sql\n",
"psql -f herd.postgres.0.58.0-to-0.59.0.upgrade.sql\n",
"psql -f herd.postgres.0.59.0-to-0.60.0.upgrade.sql\n",
"psql -f herd.postgres.0.60.0-to-0.61.0.upgrade.sql\n",
"psql -f herd.postgres.0.61.0-to-0.62.0.upgrade.sql\n",
"psql -f herd.postgres.0.62.0-to-0.63.0.upgrade.sql\n",
"psql -f herd.postgres.0.63.0-to-0.64.0.upgrade.sql\n",
"psql -f herd.postgres.0.64.0-to-0.65.0.upgrade.sql\n",
"psql -f herd.postgres.0.65.0-to-0.66.0.upgrade.sql\n",
"psql -f herd.postgres.0.66.0-to-0.67.0.upgrade.sql\n",
"psql -f herd.postgres.0.67.0-to-0.68.0.upgrade.sql\n",
"psql -f herd.postgres.0.68.0-to-0.69.0.upgrade.sql\n",
"psql -f herd.postgres.0.69.0-to-0.70.0.upgrade.sql\n",
"psql -f herd.postgres.0.70.0-to-0.71.0.upgrade.sql\n",
"psql -f herd.postgres.0.71.0-to-0.72.0.upgrade.sql\n",
"psql -f herd.postgres.0.72.0-to-0.73.0.upgrade.sql\n",
"psql -f herd.postgres.0.73.0-to-0.74.0.upgrade.sql\n",
"psql -f herd.postgres.0.74.0-to-0.75.0.upgrade.sql\n",
"psql -f herd.postgres.0.75.0-to-0.76.0.upgrade.sql\n",
"psql -f herd.postgres.0.76.0-to-0.77.0.upgrade.sql\n",
"psql -f herd.postgres.0.77.0-to-0.78.0.upgrade.sql\n",
"psql -f herd.postgres.0.78.0-to-0.79.0.upgrade.sql\n",
"psql -f herd.postgres.0.79.0-to-0.80.0.upgrade.sql\n",
"psql -f herd.postgres.0.80.0-to-0.81.0.upgrade.sql\n",

Ultimately, the CloudFormation script still fails:
CREATE_FAILED | AWS::CloudFormation::WaitCondition | herdServerWaitCondition | WaitCondition timed out. Received 0 conditions when expecting 1
  | 14:52:36 UTC-0700 | CREATE_IN_PROGRESS | AWS::CloudFormation::WaitCondition | herdServerWaitCondition

How can I help restructure this go get around this error. Happy to contribute to the project, just need some idea how I could do so.

[nateiam]

Hi Steve - Thanks for taking an interest in Herd! You are correct, it looks like we exceeded the limit for UserData. But by removing those database scripts, it killed the product install. We will get a fix for this in the near future. But in the meantime I suggest you take a look at Herd-MDL - it's the next evolution for Herd. It is a new product that includes Herd and the Herd UI but it also adds a Hive Metastore so other tools in the ecosystem can point to data managed by Herd -- and it includes an example of such a tool, a Presto cluster for interactive queries. To install Herd-MDL you can go here https://finraos.github.io/herd-mdl/docs.html and follow the instructions in the Basic Install section. This will install Herd, Herd-UI, Hive Metastore, and a Presto cluster, all integrated and ready to go. One more thing to note - the cloudformation you started working with in Herd was very minimal - all stuffed in one instance just for very light experimentation. The Herd-MDL cloudformation installs a much larger set of resources. It can get kind of expensive if you are just experimenting on your personal account. If you want to start with something a bit smaller, set the 'DeployComponents' setting to 'Herd' and it will skip the Hive Metastore and Presto cluster. Also feel free to let us know why you are interested in Herd. We always like to hear from people who are building data lakes - it's interesting to know what you are tying to accomplish, where you are in the process, etc. Hope you continue experimenting and let us know how it goes. - Nate Weisz Herd Product Manager

…

On Mon, Oct 29, 2018 at 6:07 PM stevejhall ***@***.***> wrote: When following the demo install shown here: https://github.com/FINRAOS/herd/wiki/demo-install It appears that the user data exceeds an AWS limit. This is the error I got: The following resource(s) failed to create: [herdApplicationServer]. . Rollback requested by user. CREATE_FAILED | AWS::EC2::Instance | herdApplicationServer | User data is limited to 16384 bytes (Service: AmazonEC2; Status Code: 400; Error Code: InvalidParameterValue; Request ID: 3a...) A bit of Googling seems to indicate that AWS has an upper limit of 16K By way of troubleshooting I hacked out a big chunk of the UserData, just to see if I could get past this error. That test was successful. However, I do not fully understand the impacts of doing so. This is what I deleted "psql -f herd.postgres.0.01.0-to-0.02.0.upgrade.sql\n", "psql -f herd.postgres.0.02.0-to-0.03.0.upgrade.sql\n", "psql -f herd.postgres.0.03.0-to-0.04.0.upgrade.sql\n", "psql -f herd.postgres.0.04.0-to-0.05.0.upgrade.sql\n", "psql -f herd.postgres.0.05.0-to-0.06.0.upgrade.sql\n", "psql -f herd.postgres.0.06.0-to-0.07.0.upgrade.sql\n", "psql -f herd.postgres.0.07.0-to-0.08.0.upgrade.sql\n", "psql -f herd.postgres.0.08.0-to-0.09.0.upgrade.sql\n", "psql -f herd.postgres.0.09.0-to-0.10.0.upgrade.sql\n", "psql -f herd.postgres.0.10.0-to-0.11.0.upgrade.sql\n", "psql -f herd.postgres.0.11.0-to-0.12.0.upgrade.sql\n", "psql -f herd.postgres.0.12.0-to-0.13.0.upgrade.sql\n", "psql -f herd.postgres.0.13.0-to-0.14.0.upgrade.sql\n", "psql -f herd.postgres.0.14.0-to-0.15.0.upgrade.sql\n", "psql -f herd.postgres.0.15.0-to-0.16.0.upgrade.sql\n", "psql -f herd.postgres.0.16.0-to-0.17.0.upgrade.sql\n", "psql -f herd.postgres.0.17.0-to-0.18.0.upgrade.sql\n", "psql -f herd.postgres.0.18.0-to-0.19.0.upgrade.sql\n", "psql -f herd.postgres.0.19.0-to-0.20.0.upgrade.sql\n", "psql -f herd.postgres.0.20.0-to-0.21.0.upgrade.sql\n", "psql -f herd.postgres.0.21.0-to-0.22.0.upgrade.sql\n", "psql -f herd.postgres.0.22.0-to-0.23.0.upgrade.sql\n", "psql -f herd.postgres.0.23.0-to-0.24.0.upgrade.sql\n", "psql -f herd.postgres.0.24.0-to-0.25.0.upgrade.sql\n", "psql -f herd.postgres.0.25.0-to-0.26.0.upgrade.sql\n", "psql -f herd.postgres.0.26.0-to-0.27.0.upgrade.sql\n", "psql -f herd.postgres.0.27.0-to-0.28.0.upgrade.sql\n", "psql -f herd.postgres.0.28.0-to-0.29.0.upgrade.sql\n", "psql -f herd.postgres.0.29.0-to-0.30.0.upgrade.sql\n", "psql -f herd.postgres.0.30.0-to-0.31.0.upgrade.sql\n", "psql -f herd.postgres.0.31.0-to-0.32.0.upgrade.sql\n", "psql -f herd.postgres.0.32.0-to-0.33.0.upgrade.sql\n", "psql -f herd.postgres.0.33.0-to-0.34.0.upgrade.sql\n", "psql -f herd.postgres.0.34.0-to-0.35.0.upgrade.sql\n", "psql -f herd.postgres.0.35.0-to-0.36.0.upgrade.sql\n", "psql -f herd.postgres.0.36.0-to-0.37.0.upgrade.sql\n", "psql -f herd.postgres.0.37.0-to-0.38.0.upgrade.sql\n", "psql -f herd.postgres.0.38.0-to-0.39.0.upgrade.sql\n", "psql -f herd.postgres.0.39.0-to-0.40.0.upgrade.sql\n", "psql -f herd.postgres.0.40.0-to-0.41.0.upgrade.sql\n", "psql -f herd.postgres.0.41.0-to-0.42.0.upgrade.sql\n", "psql -f herd.postgres.0.42.0-to-0.43.0.upgrade.sql\n", "psql -f herd.postgres.0.43.0-to-0.44.0.upgrade.sql\n", "psql -f herd.postgres.0.44.0-to-0.45.0.upgrade.sql\n", "psql -f herd.postgres.0.45.0-to-0.46.0.upgrade.sql\n", "psql -f herd.postgres.0.46.0-to-0.47.0.upgrade.sql\n", "psql -f herd.postgres.0.47.0-to-0.48.0.upgrade.sql\n", "psql -f herd.postgres.0.48.0-to-0.49.0.upgrade.sql\n", "psql -f herd.postgres.0.49.0-to-0.50.0.upgrade.sql\n", "psql -f herd.postgres.0.50.0-to-0.51.0.upgrade.sql\n", "psql -f herd.postgres.0.51.0-to-0.52.0.upgrade.sql\n", "psql -f herd.postgres.0.52.0-to-0.53.0.upgrade.sql\n", "psql -f herd.postgres.0.53.0-to-0.54.0.upgrade.sql\n", "psql -f herd.postgres.0.54.0-to-0.55.0.upgrade.sql\n", "psql -f herd.postgres.0.55.0-to-0.56.0.upgrade.sql\n", "psql -f herd.postgres.0.56.0-to-0.57.0.upgrade.sql\n", "psql -f herd.postgres.0.57.0-to-0.58.0.upgrade.sql\n", "psql -f herd.postgres.0.58.0-to-0.59.0.upgrade.sql\n", "psql -f herd.postgres.0.59.0-to-0.60.0.upgrade.sql\n", "psql -f herd.postgres.0.60.0-to-0.61.0.upgrade.sql\n", "psql -f herd.postgres.0.61.0-to-0.62.0.upgrade.sql\n", "psql -f herd.postgres.0.62.0-to-0.63.0.upgrade.sql\n", "psql -f herd.postgres.0.63.0-to-0.64.0.upgrade.sql\n", "psql -f herd.postgres.0.64.0-to-0.65.0.upgrade.sql\n", "psql -f herd.postgres.0.65.0-to-0.66.0.upgrade.sql\n", "psql -f herd.postgres.0.66.0-to-0.67.0.upgrade.sql\n", "psql -f herd.postgres.0.67.0-to-0.68.0.upgrade.sql\n", "psql -f herd.postgres.0.68.0-to-0.69.0.upgrade.sql\n", "psql -f herd.postgres.0.69.0-to-0.70.0.upgrade.sql\n", "psql -f herd.postgres.0.70.0-to-0.71.0.upgrade.sql\n", "psql -f herd.postgres.0.71.0-to-0.72.0.upgrade.sql\n", "psql -f herd.postgres.0.72.0-to-0.73.0.upgrade.sql\n", "psql -f herd.postgres.0.73.0-to-0.74.0.upgrade.sql\n", "psql -f herd.postgres.0.74.0-to-0.75.0.upgrade.sql\n", "psql -f herd.postgres.0.75.0-to-0.76.0.upgrade.sql\n", "psql -f herd.postgres.0.76.0-to-0.77.0.upgrade.sql\n", "psql -f herd.postgres.0.77.0-to-0.78.0.upgrade.sql\n", "psql -f herd.postgres.0.78.0-to-0.79.0.upgrade.sql\n", "psql -f herd.postgres.0.79.0-to-0.80.0.upgrade.sql\n", "psql -f herd.postgres.0.80.0-to-0.81.0.upgrade.sql\n", Ultimately, the CloudFormation script still fails: CREATE_FAILED | AWS::CloudFormation::WaitCondition | herdServerWaitCondition | WaitCondition timed out. Received 0 conditions when expecting 1 | 14:52:36 UTC-0700 | CREATE_IN_PROGRESS | AWS::CloudFormation::WaitCondition | herdServerWaitCondition How can I help restructure this go get around this error. Happy to contribute to the project, just need some idea how I could do so. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#393>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ADFd9ar0j4BMLDmTTNg9oVp4pzgydRVTks5up3wJgaJpZM4YApC5> .

[mrperson2015]

As a update, I ran through the demo install steps using herd-scripts-cloud-formation-0.88.0.jar. This issue is still very much open.

I am more interested in the data catalog piece than anything else. Our storage platform is not S3. This will catalog an event store in another system.

[ssaha01]

Nate,
Following your suggestion, I am trying to install herd-mdl through the provided installMDL.yml file. The installation is failing with the following error:

2019-07-01 12:11:23 UTC-0400	ice-poc-herd-mdl	ROLLBACK_IN_PROGRESS	The following resource(s) failed to create: [MdlStack]. . Rollback requested by user.
2019-07-01 12:11:23 UTC-0400	MdlStack	CREATE_FAILED	Embedded stack arn:aws:cloudformation:us-east-1:681636940503:stack/ice-poc-herd-mdl-MdlStack-16AJ6J361EUU6/ca1a8850-9c1a-11e9-8652-0eb49284c068 was not successfully created: The following resource(s) failed to create: [PrerequisitesPrimaryStack].

I tried to get the PowerUserAccess policy, but the link provided to get the sample policy is giving 404. Can you please help?

[nateiam]

Hi @ssaha01 -

Glad you are looking at Herd! The best course of action here is to use the Herd-MDL install as discussed in the comment above. We also recommend setting 'DeployComponents' to 'Herd' -- unless you are interested in the integrated Hive Metastore and Presto cluster.

@rongwang is an engineer from our team and she can dig into the current resource creation issue.

[rongwang0930]

@ssaha01 For better assist you, we'd like you retry the creation and send us more detail information.
1.can you retry your stack creation with Rollback on failure set to No (Rollback on failure setting is under advanced section when you create cft from console)?
2. find the actual failed nested stack inside PrerequisitesPrimaryStack, and send us a screenshot of the Events history for the failed nested stack

[dlutz2]

I found that you can get below the EC2 user data limit by removing the parts of herd.localdb.template that loads the demo data: all the curl POSTs between lines 633-729.
That allows the CloudFormation script to run but herd-app doesn't start. Tomcat logs show a permission denied exception when trying to write "velocity.log". Caused by: java.io.FileNotFoundException: velocity.log (Permission denied)
at java.io.FileOutputStream.open0(Native Method) ~[?:1.8.0_242]
at java.io.FileOutputStream.open(FileOutputStream.java:270) ~[?:1.8.0_242]
at java.io.FileOutputStream.(FileOutputStream.java:213) ~[?:1.8.0_242]
at java.io.FileOutputStream.(FileOutputStream.java:133) ~[?:1.8.0_242]
at org.apache.log4j.FileAppender.setFile(FileAppender.java:294) ~[log4j-1.2.17.jar:?]
at org.apache.log4j.RollingFileAppender.setFile(RollingFileAppender.java:207) ~[log4j-1.2.17.jar:?]
at org.apache.log4j.FileAppender.(FileAppender.java:110) ~[log4j-1.2.17.jar:?]
at org.apache.log4j.RollingFileAppender.(RollingFileAppender.java:79) ~[log4j-1.2.17.jar:?]
at org.apache.velocity.runtime.log.Log4JLogChute.initAppender(Log4JLogChute.java:118) ~[velocity-1.7.jar:1.7]
at org.apache.velocity.runtime.log.Log4JLogChute.init(Log4JLogChute.java:85) ~[velocity-1.7.jar:1.7]
at org.apache.velocity.runtime.log.LogManager.createLogChute(LogManager.java:157) ~[velocity-1.7.jar:1.7]
at org.apache.velocity.runtime.log.LogManager.updateLog(LogManager.java:269) ~[velocity-1.7.jar:1.7]
at org.apache.velocity.runtime.RuntimeInstance.initializeLog(RuntimeInstance.java:871) ~[velocity-1.7.jar:1.7]
at org.apache.velocity.runtime.RuntimeInstance.init(RuntimeInstance.java:262) ~[velocity-1.7.jar:1.7]
at org.apache.velocity.runtime.RuntimeSingleton.init(RuntimeSingleton.java:112) ~[velocity-1.7.jar:1.7]
at org.apache.velocity.app.Velocity.init(Velocity.java:74) ~[velocity-1.7.jar:1.7]
at org.finra.herd.service.helper.VelocityHelper.(VelocityHelper.java:41) ~[herd-service-0.116.0.jar:?]

Tomcat is apparently trying to write velocity.log in /root. Running Tomcat as root (very bad idea) works. Add to the cft script:
"/bin/sed -i 's/TOMCAT_USER=\"tomcat\"/TOMCAT_USER=\"root\"/g' /usr/share/tomcat8/conf/tomcat8.conf\n",

Also as far as I can tell, there is not a default user/password created for herd-ui, so it is not possible to login.

I am trying to switch to herd-mdl but our restricted environment doesn't allow creation of the required IAM entities, so having to rewrite the CloudFormation scripts to use existing entities. (which we did for the herd scripts but herd-mdl is more complex)

[TakumiHaruta]

I tried to use version 0.136.0 and could pass through User data is limited to 16384 bytes error by replacing the db upgrade script with for loop.

          "psql -f herd.postgres.0.1.0.create.sql\n",
          "psql -f herd.postgres.0.1.0.refdata.sql\n",
          "psql -f herd.postgres.0.1.0.cnfgn.sql\n",
          "for i in {1..136}\n",
          "do\n",
          "  in=`expr $i + 1`\n",
          "  v_old=`printf %03d $i`\n",
          "  v_new=`printf %03d $in`\n",
          "  psql -f herd.postgres.0.${v_old}.0-to-0.${v_new}.0.upgrade.sql\n",
          "done\n",
          "psql -f activiti.postgres.create.engine.sql\n",
          "psql -f activiti.postgres.create.history.sql\n",
          "psql -f activiti.postgres.create.identity.sql\n",
          "psql -f quartz_tables_postgres.sql\n",
          "psql -f elasticsearch.configuration.values.sql\n",

But the UserData script still failed in the end. I got 404 during wget http://localhost:8080/herd-app/rest/buildInfo.
Sorry for the Japanese output, but it means you got 404.

/usr/bin/wget http://localhost:8080/herd-app/rest/buildInfo
--2021-01-19 01:48:49--  http://localhost:8080/herd-app/rest/buildInfo
localhost (localhost) をDNSに問いあわせています... 127.0.0.1
localhost (localhost)|127.0.0.1|:8080 に接続しています... 接続しました。
HTTP による接続要求を送信しました、応答を待っています... 404 
2021-01-19 01:48:49 エラー 404: (説明なし)。

I think herd.localdb.template doesn't work anymore. I just want to know what herd is like and don't want to use herd-mdi because it requires multiple resources, so I gave up on it.