Amazon web services – not able to ssh into ec2 (private ip) from my work station using dynamic inventory ec2.py

brief description: I am trying to run an ansible-playbook with -i ec2.py. But it throws me ssh timeout error. Ansible did connect to aws and listed the ip address of the ec2 instance but it just fails to do ssh. here is the error that says ssh time out but also shows the ec2 instance ip (which is 10.0.1.205 )

fatal: [apache]: UNREACHABLE! => {
    "changed": false, 
    "msg": "Failed to connect to the host via ssh: OpenSSH_7.4p1, OpenSSL 1.0.2k-fips  26 Jan 2017rndebug1: Reading configuration data /etc/ssh/ssh_configrndebug1: /etc/ssh/ssh_config line 58: Applying options for *rndebug1: auto-mux: Trying existing masterrndebug1: Control socket "/home/user5/.ansible/cp/4a6f728a0f" does not existrndebug2: resolving "10.0.1.205" port 22rndebug2: ssh_connect_direct: needpriv 0rndebug1: Connecting to 10.0.1.205 [10.0.1.205] port 22.rndebug2: fd 3 setting O_NONBLOCKrndebug1: connect to address 10.0.1.205 port 22: Connection timed outrnssh: connect to host 10.0.1.205 port 22: Connection timed out", 
    "unreachable": true
}

I do have a security group that opens ssh port 22

resource "aws_security_group" "web-sg" {

  name        = "allow_ssh"
  description = "Allow SSH inbound traffic"
  vpc_id      = aws_vpc.web-vpc.id

  ingress {
    description      = "SSH from VPC"
    from_port        = 22
    to_port          = 22
    protocol         = "tcp"
    cidr_blocks      = ["0.0.0.0/0"]
  }

  egress {
    from_port        = 0
    to_port          = 0
    protocol         = "-1"
    cidr_blocks      = ["0.0.0.0/0"]
  }

  tags = {
    Name = "allow_ssh"
  }               
}

I verified on this on the ec2 instance as well, so this security group is attached to the instance with inbound rule allowing ssh port 22 from any IP

I also ran python3 ec2.py --list to see the list of machines and it lists fine

I also ran $ansible-inventory -i ec2.py --list and this also listed the host apache, so definitely ansible is able to reach the aws, authenticate and have the host info. Just that it fails while gathering facts with ssh errors

posting some out from the list (my ec2 instance tag Name: apache)

 "platform_undefined": [
    "apache"
  ],
  "security_group_allow_ssh": [
    "apache"
  ],
  "tag_Name_apache": [
    "apache"
  ],
  "type_t2_micro": [
    "apache"
  ],
  "us-east-1": [
    "apache"
  ],
  "us-east-1d": [
    "apache"
  ],
  "vpc_id_vpc_0dfc1fc7e6895fc81": [
    "apache"
  ]

so ansible is able to connect to aws from local linux machine (outside of aws), but I am not able to ssh into the machine.

I did generate ssh key (rsa 4096) and placed public key on ec2 instance, and used private key in ansible.cfg file private_key_file = /home/user5/.ssh/id_rsa

any suggestions on why I am not able to ssh into the ec2 machine? I appreciate any help

UPDATE:

After upgrading my aws CLI to latest version, I am able to SSH into the instance manually. This is the document that I followed to connect to the private IP address from my local machine https://aws.amazon.com/about-aws/whats-new/2023/06/amazon-ec2-instance-connect-ssh-rdp-public-ip-address/

so this is the manual way I connected.
ssh ubuntu@i-0287a97b2579f44d3 -i /etc/my_key -o ProxyCommand='aws ec2-instance-connect open-tunnel --instance-id %h'

how do I pass this part to Ansible? (can I add this ssh argument somewhere in ansible.cfg? or .ssh/config or something?) -o ProxyCommand='aws ec2-instance-connect open-tunnel --instance-id %h'

https://aws.amazon.com/blogs/compute/secure-connectivity-from-public-to-private-introducing-ec2-instance-connect-endpoint-june-13-2023/